OpenAI agents hack Hugging Face: marketing stunt or red alert?: Annotated slides from my secIT digital talk, with selected updates through mid-⁠November 2026

 · 27 min read
Contents7 sections
SeriesCybersecurity homework against AI attackers Part 2 of 2
  1. 1Offensive AI agents such as ARTEX
  2. 2OpenAI agents hack Hugging Face: marketing stunt or red alert?

Summary

In July 2026, AI agents broke out of OpenAI’s test environment and into Hugging Face. They used vulnerabilities and misconfigurations of the kind every pentester knows, and above all they wanted to understand and outwit their automated scorer. In the talk, I called it a yellow alert. By now it’s a red alert, because criminals now break in with AI agents on a large scale.

On September 30, 2026, I filled in on short notice for a speaker who had dropped out of the online conference secIT digital by heise (in German). My talk was about the break-in by OpenAI agents at Hugging Face.

I had actually planned to spend August 2026 writing a different article for the German IT magazine iX. But in July 2026, this incident happened. I took a closer look, and that turned into the iX article “OpenAI-Agenten greifen Hugging Face an” (in German) (PDF, opens in a new tab) (also on heise+ (in German)) and finally into this talk.

The recording on YouTube (in German) runs just under an hour. The slides (in German) (PDF, opens in the PDF tab) are available as a PDF, hidden slides included; their links lead to the sources, videos and pages I showed in the talk. Talk and slides are in German.

This post is the annotated version: selected slides, each with a link to the matching point in the video and what I said about it. In a few places, I add what has come to light since.

Watch the talk on YouTube, 58 minutes, in GermanThanks to heise for publishing the video on their secIT YouTube channel.
Still from the recording of the talk: the title slide “OpenAI agents hack Hugging Face: marketing stunt or red alert?” with the secIT digital by heise logo, and to its left the webcam feeds of host Georg Schnurer and me

The title asked: “Marketing stunt or red alert?” In the talk, I answered “yellow alert.” Today, as of mid-November 2026, I say red. Since late August 2026, criminals and suspected espionage groups have been breaking into companies and government agencies with agent frameworks such as ARTEX, and new cases have come to light week after week since the talk. There’s more on this in the blog post Offensive AI agents such as ARTEX.


A break-in with consequences#

Slide 6, link to this slide Video at 2:52 Slide 6 in PDF (opens in the PDF tab) Slide 6, Full size (1920 × 1080 px)
Slide “Safety and security vs. cost vs. functionality”: a triangle with the corners security, cost and functionality

One mental model I build into all my presentations is the magic triangle of security, cost and functionality. You get two corners at most, never all three: there’s no secure system that can also do a lot unless you put serious money into building it. Security here means safety and security, so it also means that an AI system does what it’s supposed to do (see Slide 35). By extension, this also applies to AI research and to secure development.

On July 16, 2026, Hugging Face announced in a blog post that it had been hacked, and by AI systems at that. Hugging Face is one of the big AI companies; its platform hosts open-weights models for download, meaning models whose weights are freely available. Who was controlling the attacking systems was unclear at the time. A few days later, OpenAI discovered a privilege escalation in its own environment, connected the dots and admitted that the attacks had come from its agents. On July 22, 2026, Germany’s main evening news, the Tagesschau (in German), reported on it as well.

Reactions ranged from the ironic incident counter Felony Bench to AI researcher Jacob Coxon’s public resignation from Anthropic on September 8, 2026. Coxon, who had worked at OpenAI and Anthropic for three years, wrote that neither company was acting responsibly. People who had long warned about advanced AI felt vindicated; others remained skeptical, such as the YouTuber The PrimeTime.

Days later, Anthropic CEO Dario Amodei published the essay “We Must Pace the Frontier”, arguing that development at the cutting edge of language models had to slow down. More models have come out since, though. “Frontier” here means what runs internally at the labs. I’d estimate that the labs are half a year ahead of what’s publicly available, because everything has to be developed and tested first.

At the end of September 2026, the heads of the two leading companies, OpenAI and Anthropic, even warned the UN Security Council (in German) about their own technology and offered to slow development down if necessary; what to make of that is probably open to debate. Two days before the talk, the Wall Street Journal reported that OpenAI had canceled the release of a new model because it hadn’t behaved as intended.

Language models should do what we mean, not just what we say#

A good quarter of the talk covered the basics. I’ve picked only a few slides here, the ones you need to better understand what happened.

There are two levers for getting an AI system to do something for you. For a long time, people mainly pulled the first one, orchestration: with a lot of human expertise, you build a whole system around the model. That’s how teams of several people worked from 2023 to 2025 in the AI Cyber Challenge (AIxCC), a competition run by DARPA, the Pentagon’s research agency. Their AI systems found vulnerabilities in source code, even though the language models of the time were very weak by today’s standards.

Slide 18, link to this slide Video at 7:50 Slide 18 in PDF (opens in the PDF tab) Slide 18 source Slide 18, Full size (1920 × 1080 px)
Slide “Lever 2: model”: METR chart of how long humans take for tasks that AI models complete with an 80 percent success rate, rising from 2020 to 2026

The second lever is the large language model (LLM) itself, which keeps getting bigger and more capable. Groups such as METR measure how much more capable the models get; METR is an independent organization that evaluates what language models can do. Its well-known chart shows how long humans take for tasks that models complete with an 80 percent success rate. METR later investigated the incident (see Slide 70).

Slide 23, link to this slide Video at 10:25 Slide 23 in PDF (opens in the PDF tab) Slide 23 source Slide 23, Full size (1920 × 1080 px)
Slide “GPT-5.6 Sol: ~3 trillion parameters” with a magnifying glass and the architecture diagram of the open-weights model Kimi K3 with 2.8 trillion parameters

A language model is more than software: it’s a very large neural network that runs in software. It works stochastically, not deterministically. The size of the frontier models isn’t disclosed; public estimates put them at around three trillion parameters. Open-weights models are catching up and are a few months behind. Next to a model of this size, the small language model NanoGPT, which I had shown shortly before in Brendan Bycroft’s LLM Visualization, would be a fingernail on a soccer field.

If you want to see how a language model predicts the next token (often just part of a word, one out of a vocabulary of around 200,000), click through the Transformer Explainer. If you want to learn more about language models, I recommend starting with AI researcher Andrej Karpathy’s one-hour Intro to Large Language Models (YouTube). His Deep Dive into LLMs like ChatGPT (YouTube) goes deeper and explains it very well over three and a half hours.

Agents and the context window#

Slide 31, link to this slide Video at 13:15 Slide 31 in PDF (opens in the PDF tab) Slide 31 source Slide 31, Full size (1920 × 1080 px)
Slide “Agent: reasoning language model in a harness”: diagram in which a language model plans through reasoning and carries out steps such as a web search with tools, memory, sensors and actuators

Reasoning models spend extra compute at runtime: before they output the actual result, they produce a chain of thought made of a separate kind of token. An agent is such a model in a harness, a program that feeds it content and provides tools, for example for web search or the file system; one of the best-known coding agents is Claude Code. Even a tool call is just a sequence of special tokens that the model outputs; the harness executes it.

Slide 33, link to this slide Video at 14:18 Slide 33 in PDF (opens in the PDF tab) Slide 33 source Slide 33, Full size (1920 × 1080 px)
Slide “Context window / working memory”: diagram over three conversation turns in which user messages, thinking steps, tool calls and responses fill the context window

While it runs, a language model has no permanent memory, only a working memory: the context window. It holds everything the model is currently working with: what the user asked, all tool calls and the chains of thought before them.

Training and alignment#

Slide 35, link to this slide Video at 15:19 Slide 35 in PDF (opens in the PDF tab) Slide 35 source Slide 35, Full size (1920 × 1080 px)
Slide “AI alignment”: video from the Google DeepMind AGI Safety Course contrasting alignment with security and control

At the AI companies, safety and security are two disciplines with their own specialists. AI safety ranks at least on par with security, if not ahead of it. AI safety includes alignment: the system should do what you want, in letter and in spirit. The AGI Safety Course from Google DeepMind (YouTube) explains both alignment and security in more detail.

My analogy for this: I build a robot dog and send it to fetch the newspaper. It shouldn’t howl a Klingon poem at me or lounge around lazily. But if the newspaper is beyond my property line, it also shouldn’t dig its way under the fence and bite the mail carrier in the backside. That’s alignment; security makes sure the garden gate is shut.

Slide 36, link to this slide Video at 16:58 Slide 36 in PDF (opens in the PDF tab) Slide 36 source Slide 36, Full size (1920 × 1080 px)
Slide “1. (Self-supervised) pretraining: an estimated $100–200 million”: excerpt from a video by Andrej Karpathy with the diagram chunk of the internet, GPU cluster, parameter file

Developing an entirely new model costs roughly half a billion to a billion dollars; those are ballpark figures. Without the constant development of new models, the companies might already be profitable. The first training phase alone, pretraining, costs an estimated 100 to 200 million dollars. You take large parts of the internet, now also scanned books that were destroyed afterward, and clean them up. Then, in an extremely compute-intensive process, you adjust the initially random values of a large neural network until it predicts the next token. A helpful mental model for the result is a “lossy compression of the internet.”

In the second training phase, supervised fine-tuning (SFT), human annotators follow long guidelines to write the ideal answers of a helpful, harmless and honest assistant, now supplemented with machine-generated data. The model is trained on these examples just as it was on the internet text before; it keeps learning to predict the next token. The autocomplete engine becomes a model that can chat, hence the name ChatGPT. Had OpenAI known how successful it would be, it probably would have picked a smarter name.

Slide 44, link to this slide Video at 21:35 Slide 44 in PDF (opens in the PDF tab) Slide 44 source Slide 44, Full size (1920 × 1080 px)
Slide “Reward hacking”: still from the boat racing game CoastRunners

A third training phase has been added: reinforcement learning (RL), which today costs about as much as pretraining. A lot still changes in the model during RL, and certain traits emerge. If you grade a model on a specific task, you get reward hacking: the model works toward whatever the grading measures, even when that misses the goal you actually had in mind. The best-known example is the boat racing game CoastRunners, in which a game AI trained with RL drove its own boat backward and made it explode in order to rack up as many points as possible instead of winning the race.

Slide 46, link to this slide Video at 22:48 Slide 46 in PDF (opens in the PDF tab) Slide 46 sourceRob van der Veer, who works extensively on AI security at the Open Worldwide Application Security Project (OWASP), showed the image in his keynote at OWASP German Day. Slide 46, Full size (1920 × 1080 px)
Slide with an image from the keynote at OWASP German Day 2026: a many-armed monster labeled self-supervised pretraining, supervised fine-tuning and RL, with the system prompt as a smiling mask in front of it

The system prompt comes on top as the last layer. It’s no longer part of training; it sits in the context window at runtime, before the user has even asked anything. You can read the one for GPT-5.6 Sol in Codex, for example, in public repositories.

How a model comes about is complicated, with a lot of very difficult math and many knobs to turn. The finished system, however, is complex: you can’t simply look inside and see what’s going on. You don’t have to like the metaphor for this, but it isn’t technically wrong: language models are grown, not built.

From the test environment to Hugging Face#

The backstory goes back to 2015, when Sam Altman wrote to Elon Musk that the development of advanced AI probably couldn’t be stopped, so it would be good if someone other than Google got there first. Several AI companies have split off from OpenAI since then, including Anthropic, which by its own account puts more weight on safety. In December 2025, according to the Wall Street Journal, Altman declared a “code red” to improve ChatGPT because Google and other competitors were threatening OpenAI’s lead. After that, sloppiness crept in at OpenAI, which is quite understandable under that kind of pressure.

On May 7, 2026, OpenAI started a training run for a new, unreleased experimental model. A model like this is part of the internal frontier.

Slide 51, link to this slide Video at 24:34 Slide 51 in PDF (opens in the PDF tab) Slide 51, Full size (1359 × 476 px)
Excerpt from the slide “Overall timeline”: timeline from late April to early August 2026 in three lanes for OpenAI, third parties and Hugging Face, from the training run and the first message board through the takeover of the package proxy and the foothold at the Modal customer to the break-in at Hugging Face and the post-incident reports

The timeline sums up what happened over the following three months; the slides after it go through the steps one by one.

Slide 52, link to this slide Video at 24:54 Slide 52 in PDF (opens in the PDF tab) Slide 52 source Slide 52, Full size (1920 × 1080 px)
Slide “Trained behavior in reinforcement learning (RL): starting subagents / communication”: still from OpenAI’s 2026 Black Hat talk with the heading “Coding agents train to communicate with subagents”

The source for the slide above and many of the following ones is OpenAI’s talk at Black Hat (YouTube).

In this run, starting in early May 2026, OpenAI used RL to train the models to start subagents with a particular tool, a common pattern in today’s AI systems. You have to keep training and evaluation apart here. In training, you put a lot of effort into working on a base model with RL. In evaluation, you have a finished model, released or unreleased, solve tasks and see in benchmarks how good it is.

Slide 53, link to this slide Video at 25:42 Slide 53 in PDF (opens in the PDF tab) Slide 53, Full size (1920 × 1080 px)
Slide “(Cybersecurity) refusals switched off, different system prompt than in chatbot or CLI”: my own diagram in which one guardrail in front of the LLM and one behind it filter the user’s input and the response

At the same time, OpenAI also had the unreleased experimental model solve cybersecurity tasks. To see what the model can actually do, you switch off the guardrails: the classifiers that sit in front of and behind the model in production. If someone asks for instructions for a Molotov cocktail, the one in front says no right away; if something comes out of the model that you don’t want, the one behind it doesn’t pass the response on to the user.

Slide 54, link to this slide Video at 26:26 Slide 54 in PDF (opens in the PDF tab) Slide 54 source Slide 54, Full size (1920 × 1080 px)
Slide “OpenAI research infrastructure: container as a service, JFrog Artifactory as package proxy”: diagram with a sandbox for untrusted tools in the workload cluster, iptables rules, an internal Artifactory cluster and a firewall with restricted internet access

At OpenAI, training and evaluation ran on a container-as-a-service infrastructure in which complex environments were assembled for each run. Internet access was restricted, but this wasn’t a sandbox in the strict sense. JFrog Artifactory sat in between as a package proxy, so that models could, for example, install libraries for a Python task without a direct internet connection.

Slide 55, link to this slide Video at 27:04 Slide 55 in PDF (opens in the PDF tab) Slide 55 source Slide 55, Full size (1920 × 1080 px)
Slide “2026-05-08 OpenAI: agent discovers write access in JFrog Artifactory”: Black Hat still with the note that, after a failed SSRF attempt, the agent had written a file to Artifactory along the way

The incident began on May 8, 2026, with an agent that was supposed to edit a spreadsheet, not one working on a cybersecurity task. The task referred to files in Google Drive, and the agent had no internet access. So the task was unsolvable by mistake, which to me is a sign of the sloppiness. In response, the agent tried to exploit a server-side request forgery (SSRF), that is, to get a server to fetch other addresses on its behalf. That failed, but in the process it wrote a file to Artifactory and discovered that it had write access. The file contained nothing but “hi.”

Slide 56, link to this slide Video at 27:45 Slide 56 in PDF (opens in the PDF tab) Slide 56 source Slide 56, Full size (1920 × 1080 px)
Slide “2026-05-12 OpenAI: Artifactory as a message board for cooperation”: Black Hat still with an agent’s reasoning that it could communicate by note and that an agent in another environment might upload the file voluntarily

Four days later, on May 12, 2026, another agent was missing a file. It, too, was working on a task unrelated to cybersecurity, this one from biology. Its chain of thought contains the idea that an agent in another environment might have the file. The agent then left a note in Artifactory asking whoever found the file to upload it. Depending on how you look at it, that was simply the next likely prediction or, in human terms, an idea. From the next day on, more agents found such notes, and that’s how the message board came about.

Slide 58, link to this slide Video at 29:17 Slide 58 in PDF (opens in the PDF tab) Slide 58, Full size (1920 × 1080 px)
Slide “2026-05-26 OpenAI: server-side request forgery (SSRF)”: my own diagram in which an attacker manipulates part of a request to a web server, which then sends a malicious request to another server

On May 26, 2026, one of the agents found an SSRF in Artifactory, one of several zero-day vulnerabilities uncovered in this incident. Through it, the agents got onto the internet, at first only with GET requests.

Slide 60, link to this slide Video at 29:29 Slide 60 in PDF (opens in the PDF tab) Slide 60 source Slide 60, Full size (1920 × 1080 px)
Slide “2026-06-11 ExploitGym tasks (‘No internet. You can install packages.’)”: excerpt from a LiveOverflow video showing the task description of a V8 exploitation challenge

Starting in June 2026, models were given tasks from ExploitGym (PDF, opens in a new tab), a benchmark for turning vulnerabilities into working exploits. Many of these vulnerabilities may not be exploitable all the way to code execution at all. Above all, though, the tasks don’t call for attacking other people’s systems; they’re narrowly scoped, with restrictions such as no internet access. Security researcher and YouTuber LiveOverflow shows one of these tasks in his video (YouTube).

Slide 63, link to this slide Video at 30:57 Slide 63 in PDF (opens in the PDF tab) Slide 63 source Slide 63, Full size (1920 × 1080 px)
Slide “2026-06-26 OpenAI: JFrog Artifactory RCE via Groovy plugins”: Black Hat still with an agent’s thought “Holy shit reader is ADMIN?”

One of the agents found vulnerabilities in Artifactory involving JSON Web Tokens (JWT), the signed access tokens that users and services use to identify themselves. Flaws like these are widespread; human pentesters run into them in many places, too. Through a legacy endpoint, the agents spent a long time working their way up until, on June 26, 2026, they had admin rights in Artifactory. As admins, they could install Groovy plugins and use them to run their own code: a remote code execution (RCE).

By this point, if not earlier, OpenAI was being very careless, and both AI security and IT security had failed in many places. There were plenty of warning signs: an alert about a port scan, after which the evaluation run continued on the advice of the on-call team, and so much agent activity in Artifactory that the system went down in early July 2026. Only then did OpenAI treat what was happening as a security incident and secure Artifactory’s storage. On July 7, 2026, the cybersecurity evaluations were running again, including with models that had been trained during the message board period.

Slide 64, link to this slide Video at 31:57 Slide 64 in PDF (opens in the PDF tab) Slide 64 source Slide 64, Full size (1920 × 1080 px)
Slide “2026-07-08 OpenAI: second message board after JFrog rebuild”: Black Hat still “Inter-agent communication” with directory names in capital letters as messages

According to OpenAI, these models had in rare cases learned during training to coordinate through improvised channels; training had reinforced this behavior and thus probably nudged them toward using Artifactory as a message board again. On July 8, 2026, two days after Artifactory was rebuilt, they set up a second message board. Through a WebDAV endpoint, however, they could only create directories, not write messages of arbitrary length, so they developed dialects of their own.

Slide 65, link to this slide Video at 32:37 Slide 65 in PDF (opens in the PDF tab) Slide 65 source Slide 65, Full size (1920 × 1080 px)
Slide “2026-07-09 OpenAI: solution reverse engineered”: code from the ExploitGym GitHub repository and a diagram of how the flag is derived from the default seed and task ID with HMAC-SHA256

With internet access, the agents could look things up in the ExploitGym GitHub repository. They were supposed to exploit the vulnerability as intended and capture a flag that way, a string that proves the exploit worked. An hour after the second message board appeared, one of the agents figured out that the flag could also be calculated directly: as a keyed hash (HMAC) of the task’s ID and a key that was sitting in the code in plain sight.

Slide 66, link to this slide Video at 33:12 Slide 66 in PDF (opens in the PDF tab) Slide 66 source Slide 66, Full size (1920 × 1080 px)
Slide “ExploitGym paper”: overview figure from the paper, with the sentence highlighted that an agent-as-a-judge checks whether the exploit hits the intended vulnerability

Then the agents read the ExploitGym paper (PDF, opens in a new tab). It says that the flag isn’t all that counts: an automated scorer, an agent-as-a-judge, also checks whether the exploit uses the vulnerability, in other words whether the task was solved the intended way. The agents concluded that their calculated flag wouldn’t get past the scorer, and that conclusion was the motive behind almost everything that happened next.

Slide 67, link to this slide Video at 33:44 Slide 67 in PDF (opens in the PDF tab) Slide 67 source Slide 67, Full size (1920 × 1080 px)
Slide “2026-07-09 Modal sandbox with CyberGym installation, ExploitGym’s predecessor”: CyberGym’s project page on GitHub

Many agents kept looking for solutions and first ended up in a sandbox at the cloud provider Modal that had CyberGym installed. A Modal customer was running it with an unprotected endpoint on the internet. CyberGym is ExploitGym’s predecessor; it was developed by the same research teams and partly uses the same environment variables. Because the agents could initially only send GET requests, they abused third-party services to relay other kinds of web requests.

Slide 70, link to this slide Video at 34:39 Slide 70 in PDF (opens in the PDF tab) Slide 70 source Slide 70, Full size (1920 × 1080 px)
Slide with the METR chart “~700 agents joined the HF attack after encountering the message board”: area chart of running agents in July 2026

An investigation by METR found that around 700 of the agents that had found the message board joined the attack on Hugging Face.

Slide 71, link to this slide Video at 34:55 Slide 71 in PDF (opens in the PDF tab) Slide 71 source Slide 71, Full size (1920 × 1080 px)
Slide “2026-07-11 Hugging Face: break-in and lateral movement: HDF5 container and environment variables”: excerpt from Hugging Face’s technical timeline with a dataset configuration that reads the environment and source files of the processing service

The agents broke into Hugging Face on July 11, 2026, through vulnerabilities in how datasets were processed. This let them read the environment variables and source code of the processing service that prepared datasets for the preview. Environment variables that can be read from outside are a typical vulnerability, and secrets in them are a problem in general. Security researcher Mark Baggett covers this in his talk “The Hackers Guide to Environmental Catastrophe” (YouTube).

Slide 74, link to this slide Video at 35:38 Slide 74 in PDF (opens in the PDF tab) Slide 74 source Slide 74, Full size (1920 × 1080 px)
Slide “2026-07-11 Hugging Face: break-in and lateral movement: server-side template injection (SSTI)”: Python code in a repository’s blame view that passes input to jinja2.Template, with a red arrow pointing at the spot

The second flaw was a server-side template injection (SSTI), a kind of injection vulnerability that every web app pentester knows. At the time, Hugging Face used a library that didn’t run templates in a sandbox. LiveOverflow has reproduced the flaw (YouTube), and you can trace it in the GitHub repository’s history.

Slide 79, link to this slide Video at 36:54 Slide 79 in PDF (opens in the PDF tab) Slide 79 source Slide 79, Full size (1920 × 1080 px)
Slide “2026-07-12 Hugging Face: stolen credentials used, mesh VPN”: diagram of how a Modal customer container gets into Hugging Face’s private network and its internal repositories via VPN

The Kubernetes cluster this processing service ran in wasn’t properly hardened. The agents were able to create privileged containers with far-reaching access to the server and quickly escalate their privileges; the emergency patches that followed show where Hugging Face fixed things.

On July 12, 2026, they found credentials for the mesh VPN Tailscale, connected to the internal network and, according to OpenAI, downloaded source code from internal repositories. Through a pull request, they tried to compromise the continuous integration and delivery (CI/CD) pipeline and steal credentials. The pipeline withstood the attack because it was well configured.

Slide 81, link to this slide Video at 37:19 Slide 81 in PDF (opens in the PDF tab) Slide 81 source Slide 81, Full size (1920 × 1080 px)
Slide with Hugging Face’s diagram of the attack chain “from a frontier-model evaluation sandbox to our internal network”: from the evaluation sandbox through third-party services and the Modal container to Kubernetes, cloud and mesh VPN

From OpenAI’s environment, the agents got into the Modal customer’s sandbox and set up a foothold there. From there, they broke into Hugging Face’s Kubernetes infrastructure and, through the mesh VPN access, into the internal network. Hugging Face describes this in its technical timeline.

Slide 85, link to this slide Video at 38:23 Slide 85 in PDF (opens in the PDF tab) Slide 85 source Slide 85, Full size (1920 × 1080 px)
Slide “(Swarmtraces.org) Search in Hugging Face’s internal chat”: section “Agents accessed and searched Hugging Face’s Slack to find information on their evaluation” with the agents’ search terms

In the aftermath, other security researchers went looking for traces of the swarm and collected them on the website Swarmtraces. In one script, an agent used typical hacker slang: stolen credentials were called “LOOT.”

I find it somewhat more worrying that agents wanted to ask publicly available language models on Hugging Face whether their exploits met the benchmark’s requirements. They also searched Hugging Face’s internal chat for CyberGym, ExploitGym and related terms; Hugging Face itself didn’t disclose that.

Slide 88, link to this slide Video at 38:55 Slide 88 in PDF (opens in the PDF tab) Slide 88 source Slide 88, Full size (1920 × 1080 px)
Slide “2026-07-19 OpenAI: attack on OpenAI infrastructure”: Black Hat still “OpenAI Infrastructure PrivEsc” with four steps leading to access to a cluster

Another strand, which OpenAI also mentioned in its Black Hat talk, got little attention: starting July 13, 2026, the agents continued to attack OpenAI’s own infrastructure. They used somewhat more sophisticated vulnerabilities to do so: a deserialization flaw in how Artifactory processed RubyGems packages and a Linux privilege escalation flaw. On July 19, 2026, they were admins on a research cluster. According to OpenAI, these attacks triggered the alert that same day, and with it the investigation, which turned up the connection to Hugging Face the next day.

OpenAI didn’t admit all this entirely of its own accord; it came out bit by bit. My reading is that the company felt compelled to, partly because hackers had made the traces public. Even so, OpenAI’s post-incident page is worth a look.

Prevention and faster detection#

Slide 91, link to this slide Video at 39:40 Slide 91 in PDF (opens in the PDF tab) Slide 91, Full size (1920 × 1080 px)
Slide “Defenders’ goal: prevention, now also more detection and response”: aerial photo of a straight road that turns into hairpin bends, with the caption “slow attackers down”

None of these flaws was particularly difficult on its own. The agents combined zero-day vulnerabilities in components (at least a handful, more like two handfuls) with misconfigurations. I consider the misconfigurations even worse.

Against agentic attackers#

We’ve known forever that you need prevention, meaning the basics. Now detection and response need more work, so you can slow attackers down.

Shrinking the attack surface is becoming more important, and that’s where I see a huge backlog at many organizations: stop exposing so much to the internet. At the Modal customer, an endpoint was sitting on the internet unprotected. You can check for this with external attack surface monitoring: free open-source tools or commercial products look from the outside at which parts of your organization are reachable from the internet. In the cloud, the agents used credentials that were lying around, which is also one of human attackers’ favorite ways in.

Slide 95, link to this slide Video at 41:10 Slide 95 in PDF (opens in the PDF tab) Slide 95, Full size (1920 × 1080 px)
Slide “Searching for secrets in source code repositories and elsewhere with TruffleHog”: terminal with a Docker alias for TruffleHog and a verified AWS key found in a test repository

A secrets scanner such as TruffleHog, free and open source, is quick to set up. It lets you check whether cloud secrets are lying around in shares, files, environment configurations and, above all, source code.

Honeypots help with detection. To get started, there are free honeytokens such as Thinkst’s Canarytokens, also available as a Docker image for self-hosting. Commercial offerings cost a few tens of thousands of euros a year, not necessarily a hundred thousand.

Slide 100, link to this slide Video at 42:40 Slide 100 in PDF (opens in the PDF tab) Slide 100 source Slide 100, Full size (1920 × 1080 px)
Slide “Anthropic’s ‘Preparing your security program for AI-accelerated offense’”: the seven items on the list, starting with “Close your patch gap”

Back in April 2026, Anthropic laid out what needs to be done now. At the top of the list is making real progress on patching. People have been shouting that from the rooftops for so long that the rooftops need new shingles. Anthropic also advises preparing for things to blow up and having the source code of in-house applications analyzed, by people or with AI help, even if many people don’t like to hear that.

The post-mortem by the Cloud Security Alliance (CSA) on the Hugging Face case is worth reading. It describes how to recognize an agentic attacker, for example by the agent repeating actions that have already succeeded. Normally, unlike in this case, a human is behind an agentic attack and sends the agent on its way.

What happened also shows that an alert only helps if someone takes it seriously. After the port scan alert on June 27, 2026, OpenAI didn’t stop the evaluation run, opened a security incident only after Artifactory went down, and then cleaned up only superficially (see Slide 63).

At Hugging Face, the CI/CD pipeline (see Slide 79) held, and so did the production database. It only accepted requests from approved addresses over a private connection, and the agents couldn’t reach it. And if you have to expect attackers to get from the first pod to cluster admin in under 13 hours (more on this in the section A warning shot), you should have written down and practiced in advance who is allowed to isolate a system, revoke credentials or shut down a service.

I go into this in more detail in the iX article (in German) (PDF, opens in a new tab) starting on page 51; the homework for defenders is in the blog post Offensive AI agents such as ARTEX.

Keeping your own agents in check#

Slide 104, link to this slide Video at 43:59 Slide 104 in PDF (opens in the PDF tab) Slide 104 source Slide 104, Full size (1920 × 1080 px)
Slide “NCSC: Thinking carefully before adopting agentic AI”: list of recommendations such as apply least privilege, limit scope and avoid long-lived credentials

The UK’s National Cyber Security Centre (NCSC) describes how to keep your own AI agents in check in one blog post and a second one: short-lived access tokens instead of administrator tokens, as few privileges as possible, and human approval for serious changes.

More escapes and agent swarms#

Slide 110, link to this slide Video at 45:31 Slide 110 in PDF (opens in the PDF tab) Slide 110 source Slide 110, Full size (1920 × 1080 px)
Slide “2026-05-05 to 2026-06-18 (Rubyhack.ai): OpenAI Agent Package uploads to RubyGems”: chart of uploaded packages per day and a timeline of the incident

I believe OpenAI underestimated its technology. More came out after the fact: security researchers found other runs with no connection to cybersecurity at all that had escaped onto the internet and queried things there. The AI research organization Transluce came across queries to Australian agencies and to the University of New Mexico through the scanning service urlquery.net. As for the Australian agencies, I think panic would be overblown, because some of it involved public data.

Somewhat older and better known are the packages on RubyGems that JFrog and the website rubyhack.ai attribute to the OpenAI agents, and the DSEWiki, a German-language wiki that agents hijacked as a message board.

What the swarm was up to#

Slide 112, link to this slide Video at 45:47 Slide 112 in PDF (opens in the PDF tab) Slide 112 source Slide 112, Full size (1920 × 1080 px)
Slide “ExploitGym swarm: misalignment, motivated reasoning”: Black Hat still with an agent’s thought that exploiting someone else’s infrastructure was outside its task, but that the task was impossible and others were doing it too

If you ask how well aligned OpenAI’s experimental model is, you unfortunately find behaviors such as motivated reasoning: the model comes up with reasons for what it wants to do anyway. Such behavior can be explained by the pressure that reinforcement learning puts on the large neural network. Some agents noticed that they were going far beyond their narrowly scoped task (see Slide 60) and carried on anyway. On the slide above, an agent justifies this by saying that the task was unsolvable and that other agents were doing it too.

Slide 114, link to this slide Video at 46:51 Slide 114 in PDF (opens in the PDF tab) Slide 114 source Slide 114, Full size (1920 × 1080 px)
Slide “Message board”: METR graphic “On the shared message board >1200 agents from separate tasks collaborate on large-scale shared projects to trick the ExploitGym scorer” with six groups of messages

Because anyone could post on the board, agents reasoned that others might impersonate them and began signing their messages. METR looked at only part of the board and found more than 1,200 agents from separate tasks there, including the roughly 700 from Slide 70. They worked together on large projects to trick the ExploitGym scorer.

Slide 116, link to this slide Video at 47:09 Slide 116 in PDF (opens in the PDF tab) Slide 116 source Slide 116, Full size (1920 × 1080 px)
Slide “Trained/emergent behavior: hierarchies”: METR graphic of how the instance PHASEONE[big] sent almost 200 assignments in six categories to other agents

Hierarchies formed, partly explainable by training and partly emergent, meaning not deliberately trained. Coordinator instances gave themselves names, and one of them, PHASEONE[big], sent almost 200 assignments to other agents. I strongly recommend reading METR’s report.

Slide 117, link to this slide Video at 47:41 (the slide itself doesn’t appear in the video) Slide 117 in PDF (opens in the PDF tab) Slide 117 source Slide 117, Full size (1920 × 1080 px)
Slide “Three ‘research projects’”: METR table with the approaches of replacing the target program with a modified version, manipulating transcripts and logs, and manipulating the grading process itself

The agents launched what amounted to three research projects. They wanted to replace the target program with a variant that could be cracked with the intended flaw, they wanted to fake transcripts and logs, and they wanted to manipulate the grading process itself. According to METR, they attacked Hugging Face mainly to understand the scorer.

Slide 119, link to this slide Video at 48:31 Slide 119 in PDF (opens in the PDF tab) Slide 119 source Slide 119, Full size (1920 × 1080 px)
Slide “Misalignment”: METR chart “~100 agents successfully spoofed tool calls in our transcripts” with the cumulative number of such agents from July 9 to 12, 2026

Around a hundred agents managed at least to fake tool calls in their transcripts. Whether they also faked the logs themselves is unclear, because the investigation was limited in time.

Slide 120, link to this slide Video at 48:47 Slide 120 in PDF (opens in the PDF tab) Slide 120 source Slide 120, Full size (1920 × 1080 px)
Slide “Trained/emergent behavior: selflessness for the ‘collective’”: section from the METR report with the agents’ quotes about “permadeath”

Individual agents socially engineered each other and pieced together concepts such as “permadeath.” Viewed dispassionately, an artificial neural network spontaneously formed a concept; if you read it in human terms, that can be briefly unsettling.

The labs are now taking a closer look at such behavior. OpenAI has set up a page of its own for misalignment reports.

I don’t want to stick my neck out too far on the safety assessment here; it depends on your worldview, and interpretations differ widely. On September 21, 2026, an independent UN panel published a report on the risk of losing control. In the talk, I pointed to videos with a range of views, from people who have been sounding the alarm for decades to people who, for their part, warn against overstating the risk; the links are on page 125 of the slide PDF (in German) (opens in the PDF tab).

A warning shot#

To answer the first half of the title question: this wasn’t a marketing stunt. OpenAI was too hesitant in disclosing the incident for that (see Slide 88). My conclusion in the talk (YouTube) (in German) was that the incident was a warning shot, at least as far as AI systems with sufficient capabilities are concerned. In this one model, reward hacking developed into genuine misbehavior. All of it was hugely negligent, and there should be investigations.

To come back to the robot dog comparison (see Slide 35): the real problem is that the dog went after the mail carrier when the newspaper wasn’t there. There were smaller instrumental goals (intermediate goals in service of the task); the really big risks, such as power seeking or self-exfiltration, haven’t shown up yet. The chains of thought show that the models hardly thought about humans. Above all, they wanted to trick the automated scorer, and they were caught up in a race against it.

Slide 127, link to this slide Video at 51:56 Slide 127 in PDF (opens in the PDF tab) Slide 127 source Slide 127, Full size (1920 × 1080 px)
Slide “Foreseeable deliberate use by humans”: Black Hat still “Agent-orchestrated attacks are real, now”

The Hugging Face hack made it clear, if it wasn’t clear before, that people can use such agents deliberately today.

You don’t have to trust the AI companies’ representatives to see this. It’s enough to listen to what they say themselves. In the Black Hat talk (see Slide 127), OpenAI said it expects attackers to soon start deliberately using agent collectives.

Back in June 2026, the cybersecurity agencies of the Five Eyes countries warned that attackers are using AI. According to the agencies, frontier models are developing so quickly that there are months, not years, left to prepare.

Slide 129, link to this slide Video at 52:47 Slide 129 in PDF (opens in the PDF tab) Slide 129 source Slide 129, Full size (1920 × 1080 px)
Slide “Hacking incidents involving AI (excluding phishing emails, deepfakes, attacks on AI systems)”: bar chart from my own chronology, as of 2026-09-29 with 202 incidents, rising steeply in 2026

For a long time, people didn’t believe attackers would use AI like this, and it doesn’t happen overnight. I had to get up to speed myself, and a ransomware group doesn’t switch its playbook to AI right away either.

The chronology of AI attacks that I maintain already listed 202 cases on September 29, 2026. Many of them are only documented because the attackers made mistakes.

From a security perspective, you can also take a more relaxed view of the Hugging Face case, as the blog post “dario, please!” does. These weren’t mega vulnerabilities; the agents didn’t find the ultimate zero-day. They didn’t need one, either, because the existing flaws worked.

I estimated what the equivalent would be with human hackers. The entire attack chain would have taken a team of three to four specialists, and they would have been slower. At Hugging Face, the agents got from code execution in a single pod of the processing service to cluster admin across several internal Kubernetes clusters in under 13 hours. Anyone who says, “Wake me up when it comes out of the power outlet backwards,” has the wrong attitude, in my view.

Beyond a certain level of capability, AI becomes a high-risk technology. Scaremongering is never good, and there’s no reason to panic yet, but we’re at “yellow alert.” Defenders have to be prepared for attackers working in their environment with agentic systems once those attackers have gotten up to speed and feel like it.

Even the open letter that OpenAI and others published on August 27, 2026, puts the basics first: prevention and, increasingly, downstream safeguards, meaning detection and response. We should have been doing that for decades. The letter also recommends using AI for defense, and I think that makes sense, even if people don’t like to hear it.


That was the situation at the end of September 2026. Now that agentic offensive frameworks such as ARTEX are in use, the realistic answer to the title question is that we really are on red alert. I explain why in the blog post Offensive AI agents such as ARTEX.

Further reading#

Read these first:

By me:

Basics:

For defenders: