The AI Security Comfort Blanket
There is a pattern we see in nearly every organization that deploys AI-powered security tools. It goes like this:
The CISO presents to the board. "We have deployed Darktrace across our network. It uses unsupervised machine learning to detect anomalous behavior in real time. Our network is now monitored by artificial intelligence."
The board is satisfied. The budget is approved. Everyone moves on.
What nobody asks is the question that matters most: does it actually work in our environment? Against our threat model? With our configuration?
The assumption is that AI equals covered. That is the new blind spot.
How AI Detection Actually Works
To understand why blind trust is dangerous, you need to understand how AI-powered NDR tools work under the hood. Darktrace, Vectra, ExtraHop and similar platforms use machine learning models to build a behavioral baseline of your network. They observe traffic patterns over weeks, learn what normal looks like and then flag deviations from that baseline.
This approach is genuinely powerful. It can detect novel attacks that signature-based tools miss. It can identify insider threats, compromised credentials and lateral movement patterns that static rules cannot capture.
But it has structural requirements that are often overlooked:
- Training data quality: The model is only as good as the data it trains on. If the training period includes attacker activity (which happens more often than you think), the model learns to treat that activity as normal.
- Baseline maturity: A new deployment needs two to four weeks of observation before detection accuracy reaches useful levels. During that window, coverage is limited.
- Environmental assumptions: The model assumes the network topology, traffic patterns and device population remain relatively stable. Major changes invalidate portions of the baseline.
The New Device Problem
Every time a new device joins your network, your AI security tool faces a cold start problem. It has no behavioral history for that device. It does not know what normal looks like for that specific endpoint.
Consider the attack scenario: an attacker gains physical access to your office (through social engineering, a compromised contractor badge or an unlocked server room) and plugs a small device into an open network port. That device has no baseline. For the first days or weeks of its presence, the AI model is learning what it does rather than flagging what it does.
If the attacker's device behaves consistently from the moment it connects, even if that behavior includes reconnaissance, lateral movement and data exfiltration, the AI model may incorporate that behavior into its baseline as "normal for this device."
This is not a theoretical concern. It is a documented limitation of behavioral analysis systems. The cold start window is a structural vulnerability that attackers with physical access or supply chain compromise can exploit.
The Slow Attacker Advantage
Behavioral anomaly detection works by identifying statistical outliers. An employee who usually transfers 50MB per day suddenly transferring 5GB triggers an alert. A workstation that has never connected to a particular server suddenly making hundreds of connections triggers an alert.
But what about the attacker who transfers 60MB per day? Or 70MB? The one who increases their activity by 5% each day, gradually shifting the baseline while staying within the statistical noise margin?
Slow-moving attackers are the natural predator of AI detection systems. They operate below the anomaly threshold, making incremental changes that the model absorbs into its evolving baseline. By the time the attacker is exfiltrating significant volumes of data, the model has adjusted its definition of "normal" to include that behavior.
This technique, sometimes called "low and slow" or "baseline poisoning," is well understood in the offensive security community. It is precisely the kind of technique that ShadowTap validation is designed to test.
The Seasonality Trap
Networks are not static. They have seasonal patterns, project cycles and periodic events that create temporary traffic spikes. Month-end financial processing generates different traffic than a normal Tuesday. A product launch creates different patterns than a quiet sprint. Holiday periods see different user populations than regular operations.
AI models struggle with seasonality. If the model was trained during a quiet period, a legitimate business spike can trigger false positives. If it was trained during a busy period, quieter times may cause the model to miss low-volume attacks that would have been visible against the normal baseline.
Security teams that experience high false positive rates often respond by raising detection thresholds, which directly increases the window for real attacks to go undetected. This creates a feedback loop where the AI tool's limitations drive configuration decisions that make its limitations worse.
What AI Security Tools Are Good At
Despite these limitations, AI-powered detection tools provide real value. They are genuinely effective at:
- Detecting bulk data exfiltration from established devices
- Identifying compromised credentials when usage patterns change dramatically
- Flagging new connections to known-malicious infrastructure
- Detecting ransomware encryption behavior (mass file modifications)
- Providing network visibility and asset inventory
The problem is not the tools. The problem is the assumption that the tools handle everything. They handle a significant portion of the threat landscape. But the portions they miss are exactly where sophisticated attackers operate.
The Solution: Trust but Verify
The answer is not to abandon AI security tools. The answer is to validate them. Treat your AI detection platform the same way you treat any critical system: trust it to do its job, but verify through independent testing that it is actually doing it.
At Sherlock Forensics, ShadowTap validation tests your AI detection tools against the specific techniques they are most likely to miss. We test the cold start window, the slow escalation path, the encrypted tunnel blind spot and the identity rotation gap.
Your AI security tools are a valuable layer of defense. But they are one layer. A complete defense requires red team validation to confirm that every layer works as expected.