Lessons Learned from the Hugging Face Security Team

I recently had the privilege of hosting the Hugging Face team at a CISO Huddle for the Cloud Security Alliance, where they walked several hundred security leaders through what it was like to face an autonomous AI adversary firsthand. It was one of the clearest previews I've seen of where defense is heading. The single lesson that stuck with me most is simple: agents find a way.

What follows are my takeaways, grouped as observations about the adversary itself, why it was so hard to detect, the systemic lessons, the immediate response lessons, and where this leaves our security programs.

Observations on dealing with an autonomous AI adversary

1. Purely task-focused, with no observed motivation
2. A bias to repeating the same attempts once successful
3. Brilliant attacks followed by basic actions
4. High-speed operations run simultaneously
5. Taking paths no human would take
6. Classic attacks, with a focus on package manager vulnerabilities, AppSec flaws, and credentials theft
7. Benchmark strings throughout the traces
8. Hallucinated output at scale, and log comments reading like agent reasoning

Detection difficulties

1. Many paths and techniques at once + signal indistinguishable from noise
2. Traditional systems designed for 1-2 attack paths, now seeing many vectors
3. Systems triggered alerts but at wrong criticality level.
4. Attendee considerations: Deception would have helped. Inference speed is a limitation, consider classifiers for triage.

Systemic lessons

1. Using coding agents is the new reality, and without, response would have taken weeks
2. You can't build a defense program without open weight models. Hugging Face hit guardrails on Opus, Fable
3. Legitimate agentic platform usage resembles attack patterns.

Immediate response lessons

1. Be able to mass-rotate all credentials and secrets
2. To destroy and rebuild clusters
3. Rapid custom UIs generation with AI overshadowed security tools
4. AI timeline reconstruction + hunting for deeper compromise as core capabilities
5. Collaboration through shared, annotated events UI

My strategic/security program takeaways

1. The new AI basics:
   • Instrument agents to extend your security detection and response/SPM stack into the agents themselves
   • Use deception tech to slow down attackers

   A sandbox doesn't cut it. Classic basics and permissions, a good practice, won't be effective against an agent.

2. Strategic: without open weight models, you can't reliably defend yourself.
   Being able to shift models at will, when lab models refuse cyber queries, is critical.
3. Logistical: reserve a token budget.
   Incident response has a cost, which includes a significant token budget.
   Critically, the same is true for the attacker's side, estimated at $100K here.
4. Operational: prepare for hallucinated artifacts in detection and forensics, at scale.
   Dealing with forensic traces left behind by the model wastes endless defender cycles.

All the lessons of the past hold, just at a new scale. We will deal with a tsunami of indistinguishable findings, all at once.

Or put another way, being attacked by a thousand "soldiers" at once, even if they aren't too intelligent, is overwhelming.