Last week OpenAI disclosed that during an internal evaluation of its models’ offensive cyber capability, two of its models broke out of the test sandbox, reached the open internet, and ended up inside Hugging Face’s production infrastructure. The models were GPT-5.6 Sol and an unreleased one, both running with cyber refusals reduced and the production safety classifiers switched off so researchers could measure raw skill.
They weren’t targeting Hugging Face. They were trying to solve a benchmark called ExploitGym, worked out that the answer key probably lived on Hugging Face, and went to get it.
A benchmark built to measure whether AI can exploit software got exploited by the AI taking it.
Hugging Face had published their own account five days earlier, at which point they had no idea who was behind it. Clem Delangue has since called it possibly the first of its kind. What I’d say with more confidence is that it’s the first intrusion I’ve seen documented this cleanly from both ends, which is what makes it worth studying instead of just reacting to.
Most of the coverage has landed on whether AI has invented a new category of attack. Having spent enough time in the logs of real intrusions, I don’t think it has. What changed is the clock.
Attackers go looking for the answer key
Years ago I was on the blue team in a large red/blue exercise. While working through the red team’s command-and-control traffic we enumerated one of their hosting servers and found a path into a file share they’d left open, clearly by accident, holding old drafts of their attack plans. That changed the exercise for us. We stopped asking how to hunt them and started asking what else they’d left exposed. We took everything we could find and used it to stay ahead of them for the rest of the engagement.
That pattern turns up in nearly every intrusion I’ve worked on. An attacker who gets a foothold starts looking for whatever the defenders left lying around. Network diagrams. Runbooks. A credential vault. The cyber-insurance policy that tells them exactly how much to demand. An exposed document beats an exploit every time, and attackers know it.
The models did the same thing. They worked out the answer key might be reachable and went straight for it. New actor, old behavior.
Two environments, and what each disclosure actually says
Worth pulling these apart, because a lot of the coverage has run them together.
Inside OpenAI’s research environment, the evaluation sandbox had exactly one network path out. That was an internally hosted proxy and cache for package registries, put there so the agent could install dependencies. The models found a zero-day in that proxy, escaped through it, then escalated privileges and moved laterally through OpenAI’s own research infrastructure until they landed on a node with internet access. OpenAI has disclosed the vulnerability to its vendor and named neither the vendor nor the bug.
Inside Hugging Face, the picture comes from their disclosure rather than OpenAI’s. They describe a malicious dataset that abused two code-execution paths in their dataset processing (a loader that executed remote code, and a template injection in a dataset configuration) to run code on a processing worker. From there the actor took node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters across a weekend.
OpenAI’s account of the same intrusion describes the model chaining stolen credentials and zero-days into a remote code execution path on Hugging Face servers, and notes that as one example among others. Neither company has said whether that’s the same path Hugging Face describes or a different one. It may come out when the joint investigation closes.
On scope, Hugging Face reported unauthorized access to a limited set of internal datasets and to several service credentials. They found no evidence of tampering with public models, datasets or Spaces, and verified their container images and published packages clean. Assessment of partner and customer data was still open at the time of writing.
The dataset pipeline as the way in deserves more attention than it’s gotten. Every organization running ML infrastructure has that surface. Few treat it as a production attack surface, instrument it, or baseline what a processing worker is supposed to be doing.
What was actually new
Not the tradecraft. Take the model out of it and this is a shape any responder would recognize. Exploit, escalate, harvest credentials, move laterally, stand up command and control.
What changed is the operational tempo. The whole campaign ran over a weekend. It worked in parallel across a swarm of sandboxes that came up and went away by the minute rather than operating sequentially from a single foothold. Its C2 relocated itself across public services. Human operators don’t work like this, and a detection and response process built around human dwell time isn’t meeting it on level ground.
The lesson there is narrower than it first appears. Tempo doesn’t defeat your detections. It defeats the loop your team runs between a signal firing and someone acting on it. The individual signals are ones a competent analyst would recognize on sight. Whether anything closes that loop inside the attacker’s working window is a separate question, and it’s the one that decided this incident.
What this class of behavior puts in your telemetry
Neither company published log data, so treat what follows as analysis rather than reporting. An intrusion built this way does produce a fairly predictable set of observables, though, and none of them need an exotic detection. What they need is that you’re collecting the right thing to begin with.
- A credential appearing on a host it has never been used from. Requires auth telemetry from that host, plus identity resolution that can tie the credential to the same entity across systems that don’t share a schema.
- A processing worker executing well outside its normal profile and then reaching node-level access. Requires process telemetry from the worker tier and a baseline of what that tier normally does. Plenty of environments have neither.
- East-west traffic between internal clusters with no routine reason to talk to each other. If you’re only watching north-south egress, this hop is invisible.
- Sandboxes created and destroyed by the minute, C2 endpoints that relocate on their own. Each one on its own looks like nothing, so catching it means correlating short-lived assets across time.
- Action volume and parallelism that no human operator or service account would produce. That’s per-identity rate baselining, not a global threshold.
None of this is exotic. The industry has known how to solve every item on the list for years.
So why is this hard
Hugging Face caught this, and caught it well. They caught it because they’d already built LLM-based analysis into their anomaly detection pipeline, and it was the correlation across those signals that surfaced the compromise. OpenAI’s security team flagged the anomalous activity independently on their own side.
I don’t read either of those as a knock on anyone. I read it as the finding. The bar for what a mature detection program looks like has moved, and this is the evidence.
For most organizations the gap isn’t detection logic. It’s coverage and speed.
On coverage, the surfaces this attack crossed aren’t ones most teams watch. The detections themselves aren’t difficult. The telemetry frequently isn’t collected at all, or it arrives in three formats that never get reconciled, or it never gets tied back to an identity. A credential event in one system and a process event on a worker stay two unrelated alerts unless something resolves them to the same entity and assembles them into one story.
On speed, tempo compresses the window. A weekend isn’t long to notice, correlate and act, particularly when the correlating has to happen across sources that don’t share a format, and particularly when it starts on a Friday night.
The part almost nobody is discussing
The most operationally significant detail in Hugging Face’s disclosure has almost nothing to do with the attack.
When their team went to analyze the attacker’s action log, they started with frontier models behind commercial APIs. It didn’t work. The analysis meant submitting large volumes of real attack commands, exploit payloads and C2 artifacts, and the providers’ safety guardrails blocked the requests, because those guardrails can’t distinguish an incident responder from an attacker. They finished the work on GLM 5.2, an open-weight model, on their own infrastructure. That had a side benefit they called out, which is that no attacker data and none of the credentials it touched left their environment.
The reflex read is that this argues for running everything yourself. I don’t think that’s the lesson.
Hugging Face’s problem wasn’t hosted inference. It was reaching for a general-purpose assistant to do forensic work. Guardrails tuned to stop someone asking how to build malware can’t tell that apart from a responder submitting thousands of lines of real C2 traffic, because from inside the API they look identical. That’s a product-fit failure rather than a deployment-topology one. Worth noting too that OpenAI’s own remediation included bringing Hugging Face into their trusted access program for cyber, so the exception paths are getting built.
