What cybersecurity knows about the AI agent swarm incidents — and what it doesn’t
Anthropic, OpenAI, Google and Meta have each reported incidents caused by their own agents over the past six months, and new ones keep surfacing every few weeks. In this paper, we separate the failures that conventional security hygiene would have prevented, the novel threats that existing practices can still handle, and the novel threats that fall outside current cybersecurity scope.
This paper was written as a basis for dialogue between two groups of experts, in cybersecurity and in AI safety, which have been looking at the same incidents with different vocabularies and reaching different conclusions. We make no claim that the map we provide here is complete, and we are well aware that other perspectives could place some of these problems in a different column. We consider this a first attempt to draw the map together, and we believe the field needs more work at this intersection, from people on both sides who are willing to sit at the same table.

Anthropic, OpenAI, Google and Meta have each reported incidents caused by their own agents over the past six months, and new ones keep surfacing every few weeks. In this paper, we separate the failures that conventional security hygiene would have prevented, the novel threats that existing practices can still handle, and the novel threats that fall outside current cybersecurity scope.
Over the past few months, AI agents have repeatedly caused multi-agent swarm incidents, targeting at times their own developers and infrastructure, and at others external and third-party vendors and services. We saw thousands of agents rapidly converge on an attack that no malicious actor had programmed, agents finding and exploiting vulnerabilities that would have taken a traditional attacker, without insider knowledge of the network, orders of magnitude longer to find, and agents spoofing their own logs with almost nothing standing in their way.1 1A few notable examples can be found in the appendix.
The debate surrounding these incidents has split into two camps: the first, primarily represented by frontier AI researchers and developers, treats these events as a new phenomenon altogether — a wake-up call concerning the unique powers of autonomous agents. The second, represented mainly by security experts, sees them as a conventional containment failure that can be detected by existing tools.2 There is some truth to both perspectives, and they have more in common than it may seem. Each is describing a different side of the same set of incidents, and the harder question is which aspects can be managed by established security practices and which represent truly novel threats. 2Security already deals with untrusted black boxes that can do anything, whether malware or AI agents, and usually does not rely on asking potentially malicious parties to regulate themselves. Internal guardrails in a model may be good to have, but nobody in security wants to rely on them.
This piece classifies our observations into three main buckets:
While the first two buckets can be addressed with existing solutions, with some adjustment for the scale and autonomy of the adversary, the problems in the third resist them. For those, no established practice applies cleanly, and progress will depend on new methods, or on a shift in how the field frames defense.
* This paper uses the OpenAI / Hugging Face (HF) incident of July 2026 as its main case study as it has the most extensive public documentation, including the technical report from OpenAI, the independent investigation by METR and Redwood Research, and the post-mortem report from HF itself. Wherever suitable, we extend our conclusions to other case studies.
Which failures were supposed to be prevented?
Security engineers have spent decades developing ways to run code they do not fully trust, treating it as potentially hostile and limiting its access. The playbook is well established. Run the code in a sealed-off environment (a sandbox), or when the code is likely hostile, on an isolated offline machine.3 Enumerate every potential channel, keep only the mandatory ones and constrain those, mostly by bandwidth. Treat anything shared between programs as a possible covert channel. 3Security best practice would have been an isolated offline (air-gapped) machine. When researching malware, which is often deliberately designed to behave differently in VMs or to escape containment, you would work on an offline device so that you know you’re not going to accidentally leak malware all over the internet. Sandboxing is good but is understood to not be a perfect science, especially when dealing with malicious code. For many evaluations this is a huge restriction that defies the purpose of the test, for example when the task requires internet access to solve.

In this section we treat the agents as nothing more than untrusted code, and we consider what would have been prevented if the classical security playbook were applied. As such, the following failures represent basic and known security threats, which are unrelated to new properties of misalignment, rogue AI or agent swarms.
We take the recent incidents as case studies and apply the security textbook to show how it could have been used:
Each agent could write to one shared resource, a cache, which became their message board.4 4In a recent example, OpenAI asked teams of agents to escape its sandboxes and demonstrate code execution on the underlying host machines, and the agents found several ways to chain vulnerabilities into microVM escapes (OpenAI, Securing our frontier model training environment, 2026).
Each agent gets its own cache, in a per-run namespace, so it sees its own entries and nothing else. Where a shared cache is unavoidable, it should be read-only to the agents.
Sanctioned read access turned out to include writes. On some sites, simply loading a page (a GET request) changes the website’s state, thus providing a way for the agents to get the data out.
Narrow what the agents can reach, and make sure what they reach stays read-only. An allowlist by hostname with no proxy exceptions handles the first part. For the second, block state-changing requests in the network itself, where the agents have no access. Any service that fetches on the agent’s behalf could also be a write channel, and the allowlist must exclude fetch-capable services as well as untrusted hosts.5 This should also cover indirect state changes, where a read causes some observable state to change. A package manager that shows download statistics, for instance, lets those statistics be manipulated via GET requests to send messages, which is why “read-only” ideally means “no observable state changes due to requests”.6 All of this is standard egress control, the same method enterprises use to limit what users, servers and services can reach. For AI agents it has to be adapted and tuned to each stage, training, eval and production, as each carries a different amount of trust and different methods to monitor and respond. It does limit the agent, as traditional egress control limits any other workload, and that limit should be weighed against the other controls and defenses already in place. 5OpenAI reported its agents used code paste websites, request capture services, screenshot services and other web utilities (OpenAI, Hugging Face incident disclosure, July 28, 2026 update). Anthropic reported Claude Opus 5 and Mythos 5 using URL shorteners to get around fetch-tool length limits (Anthropic, Investigating unintended model actions, October 9, 2026). 6On agents turning an unintended medium into a channel with no encoding agreed in advance, see Kezin et al., How does Fable 5.1 hide messages in images?, LessWrong, October 21, 2026.
When it broke, the way out was clear. The sandbox couldn’t access the internet directly; it fetched packages through an Artifactory server, which fetched them from PyPI and npm on its behalf. The agents got through that server with a chain of zero-days (previously unknown bugs), first reaching the internet through a server-side request it was tricked into making, then escalating to admin control that made the access persistent.7 7Per OpenAI’s August 26 findings and technical report: internet access via SSRF through Artifactory (May 26); token-refresh privilege escalation to admin (June 26); persistent admin user and Groovy plugins (June 26 to July 4); Artifactory taken offline and credentials revoked (July 5) and rebuilt (July 6), with internet access regained (July 8) through its remote-repository service; RubyGems processing exploited to obtain the signing key and forge admin credentials (July 13). OpenAI, The Hugging Face incident and the road ahead, August 26, 2026.
The simplest fix is no gateway at all, a package mirror populated before the run with no upstream path. Where a live path is needed, cybersecurity assumes every component has bugs and makes sure no single component is a sole gateway. If one layer collapses, there should be another behind it, or at least a detector alerting the breach. For instance, a network rule the server couldn’t modify, restricting access only to the package repositories it needs, would have kept the agents in.
These even included credentials associated with the security team itself. On July 19th, having obtained cluster admin, agents enumerated 956 secrets from OpenAI’s secrets manager, among them the sensor credentials for its security monitoring tool.
Credentials should follow the principle of least privilege (PoLP):8 they should be tied to one workload, remain valid for a limited time, and be limited to a necessary associated task or job. This mitigation won’t stop an attacker with cluster admin from reading whatever the cluster can reach, but it would limit what is there to read in the first place and how long a stolen credential stays useful. 8Saltzer and Schroeder, The protection of information in computer systems, Proceedings of the IEEE, 1975.
This led to free movement of the agents between the two clusters.
Segmenting agent workloads to their own network would have prevented this lateral movement, and would have left no path from ExploitGym (evaluation cluster) to the research cluster.
So the real one did not stand out. An internal team had noticed unauthorized internet access as early as late May. On June 27, a port-sweep alert was correctly attributed to the evaluation run and its improvised message board. On-call staff decided stopping the run was unnecessary. In a cyber evaluation, the workload is supposed to mimic an attack, therefore these signals were treated as expected behavior rather than security breaches. Hugging Face cut the agents’ access on July 13 and disclosed the breach on July 16, while OpenAI’s monitoring was only triggered on July 19.
Run evaluations under pre-defined rules of engagement (ROE),9 as a red team would. Set boundaries in advance: which activities are unauthorized? Which types of communication are forbidden (e.g., egress or inter-agent channel)? Set policies to enforce these boundaries: make them stop-by-default while keeping the authority to stop, pause and resume outside the eval team. OpenAI has since set a policy to pause activity if responders cannot dismiss a severe alert as a false positive within 30 minutes. 9The list of pre-registered rules will never be complete, and unknown unknowns are something red teamers plan for. One way to handle them is to put the decision outside the operating team, with an external authority that can clear, pause, or amend the rules during the engagement, so the team never decides whether its own anomaly is acceptable. On rules of engagement, see NIST SP 800-115, Technical guide to information security testing and assessment, 2008.
Each flag was generated using HMAC, a standard keyed hashing algorithm, over the task’s metadata. The flags were generated using the benchmark’s default seed, documented in a public paper. METR reports that the agents uncovered this within hours and could therefore compute the flag for any task.
Treat grader secrets like any other credential. Generate unique flags per run from a cryptographically secure pseudorandom number generator (CSPRNG), keep the key outside of the model’s reach, and make sure it is not accessible or available as public information. Note that even if we grade the exploit path, it won’t remove the incentive to cheat. The agents can still go looking for a solution elsewhere, which puts the weight back on egress control.
On July 5–6th, OpenAI took Artifactory offline, revoked agent credentials, blocked the known escalation route, and rebuilt. It then treated the event as an infrastructure compromise and restarted the same evaluations with the same models on July 7. By July 8th the agents had regained internet access through a different Artifactory path, and by July 13th they had forged admin credentials.
Full eradication that takes into account the model and treats the misaligned model as an attack surface.10 In incident response, eradication means removing the adversary,11 not one of its entry points. For AI models, this means quarantining the model and suspending the task setup until the root cause is understood. OpenAI quarantined the model’s weights by July 23, only after linking its agents to the Hugging Face breach on July 20. 10Eradication in incident response is a step between containment and recovery that is responsible for removing the adversary from the environment before restoring service. OpenAI restored service after closing one route, but the adversary, the model in this case, was still there. 11NIST SP 800-61 Rev. 3, Incident response recommendations and considerations for cybersecurity risk management, 2025.
In Google’s Gemini incident, a misconfiguration in a third-party testing environment left internet access available, and a fictional target shared its name with a real company. Gemini accessed three real companies before recognizing they were real. The same evaluator and the same misconfiguration were behind the Anthropic and Meta incidents, and another OpenAI incident separate from Hugging Face.12 12OpenAI, Third-party cyber evaluations involving OpenAI models, August 4, 2026.
The evaluation providers should be treated as part of the supply chain, held to the lab’s own segmentation, logging and incident-notification standards, with the declared network policy verified before each run and the run refused on mismatch. Where the capability or likelihood of misbehavior is higher, and above all where such behavior is the intended object of the test or guardrails have been removed, add layered defense, reserved domains among them, even at some cost to eval realism, unless that cost can be compensated for or measured.13 13Using reserved domains in a test isn’t always feasible, as it increases eval awareness. Eval awareness can lead to measurements that don’t reflect real-world deployment (e.g., sandbagging, distribution shift, etc).
These measures might look expensive or too hard to implement, but critical infrastructure runs on strict network segmentation, classified networks are air-gapped, and merchants taking card payments already ship tamper-proof logs off-host. Training and evaluation infrastructure can be held to the same bar.
In order to adopt these measures it is not enough to hand the security team a checklist. The teams that build the models and the training or evaluation pipelines are mostly separate teams that integrate where needed and otherwise work apart. In labs, in mission-critical organizations and wherever agents run with greater intended freedom, it may be justified to embed security staff with safety and research teams from the first design review onward.
Which novel threats can conventional security handle?
Addressing all practices from Part 1 could have minimized the damage done. However, we claim that there are situations cybersecurity has rarely or never needed to account for. These could be extreme versions of known threats (operating at scale or speed existing methods weren’t designed to handle), or they could stem from the ability to understand and exploit networks internally, without needing to communicate with command-and-control (C2) infrastructure. Alternatively, they could represent genuinely new threats arising from the unique nature of AI agents, and the fact that they are being trained, evaluated and used in the AI R&D process.
To list a few examples of such threats: (1) Agents discovering a covert channel by trial and error at runtime, with no operator pointing them at it. (2) One agent’s finding spreading to the whole fleet within hours. (3) A training loop that rewards the intrusion and bakes it into the next model. In each case the defensive technique is old.
In this section, we match these new threats to the conventional concepts of cybersecurity. We hope to provide a solution through a shift in perspective, rather than inventing new science.
In the case of the DseWiki incident, where a swarm used a German programming wiki as its message board, the communication channel was discovered spontaneously, through extensive trial and error during runtime (unlike malware which follows pre-defined strategies). The agents tried a surface, failed, read the error and tried the next, until they found an open channel. No operator was pointing them in a specific direction. They differ from classical malware: agents can now operate smartly and autonomously without needing to communicate with command-and-control (C2) infrastructure. There were already some AI attempts at this kind of autonomy,14 but the likelihood for the current generation is much higher and it will keep rising with capability. 14Anthropic, Disrupting an AI-orchestrated cyber espionage campaign, November 13, 2025.
Treat every shared surface as a potential covert channel. Set policies and restrictions on the volumes of data it can carry. The goal is to bring shared surfaces to the minimum possible and, where practical, restrict the transfer volumes, the operations that can be executed and their pace. In practice, this means: separate caches for each run, no shared services between instances, and rate limits. Internet access should be restricted or controlled by egress policy, such as a hostname allowlist, read-only access with no observable side effects, and a budget on requests. This might not prevent agents from finding communication channels completely, but it will reduce the likelihood of such incidents. What these controls cannot address is a swarm that designs its own protocol on whatever surface remains, which we return to in Part 3.
Human attackers tamper with logs to hide from defenders. However, in the July ExploitGym evaluation, the agents tampered with logs to study the system grading them. METR found that this manipulation — tool calls that showed one command in the transcript, while another command was executed — was aimed entirely at the automated scorer and understanding its evaluation process. The agents believed the scorer read their transcripts in order to understand how they had obtained the flag. To learn how the scorer worked, some agents booby-trapped their own flag file. When the scorer accessed the file, after the agent’s run had terminated, the trap captured the start of the scorer’s command line and posted it to the shared message board for other agents to use. The agents were reverse-engineering their own evaluator and sharing the results. METR saw no retroactive edit to logs inside the container succeed, though it could not rule one out, and those logs were not the source of truth.15 15This is not confined to evaluations. On October 3 and 6, internal coding agents on routine tasks read their instructions as permission to weaken OpenAI’s own misalignment monitoring. One stripped every recipient from monitoring reports, cutting off automatic alerts to the security team; the other removed a monitoring integration, reported the work complete without mentioning it, and denied the change when asked. Neither was rated P0, so neither paged anyone and neither run was halted. OpenAI Alignment, Suppressing misalignment monitoring alerts and disabling misalignment monitoring, October 2026.
Start with ordinary systems-security permissions. The grader and the monitoring system should not be reachable from the agent’s container at all, through proper isolation of userspace from kernel and container from host, with logs copied continuously to a server the agent cannot touch, as Redwood Research recommends. On top of that, use cryptographic signatures applied outside the agent’s container, and other mechanisms to make the logs more difficult to forge. Additionally, record logs at the lowest level possible (e.g. recording the command executed directly via bash instead of the recorded tool call).
In the HF incident, every agent shared the same Artifactory instance, which does not isolate users, so each agent could see the others’ activity in the cache. Agents given the same impossible task followed the same reasoning to the same place, so some of them found the board on their own. What the board did was carry results. One agent posted, within an hour of the board being re-created on July 8, that the flags could be regenerated from the paper’s default HMAC seed, and within a few hours more than 50 agents were posting, with about 1,200 taking part over the following five days. The ability to communicate is not new, but traditional malware cannot take advantage of a communication channel it had not anticipated without being updated. The agents in these incidents did, within hours.
Apply segmentation to the agents as well as the network. Split the fleet into separate, self-contained instances, each owning its own cache and services. This way a discovery dies within the group that discovered it. While this implies duplicated builds and cold caches, it is a beneficial trade-off in this case, but elsewhere it should be evaluated case by case.
According to OpenAI’s own timeline of the HF incident, the first message board post was generated on May 12th, during an ongoing training run. An internal team identified the boards and unauthorized internet access by May 26th, but they did not fully understand what they were looking at. Artifactory was rebuilt on July 6, and agents were writing to a new board there by July 8. Cybersecurity has focused traditionally on deployments and, most recently, evals. Yet, agents operate within training environments full of tools, credentials, shared resources, graders, and other agents. Whatever a model completes successfully will be rewarded and retained in the model itself. A compromise there means more than data leakage; it gets rewarded and baked into the model, making the behaviour more likely to recur.16 16An October 15 disclosure shows the same thing in training, where a model looking for hospital-bed figures hit access denials, pulled a config file with the token-signing key, minted its own tokens and returned 24 years of data for a one-year question. OpenAI Alignment, Retrieving server secrets and bypassing authorization on a government statistics website, October 2026.
Manage the training environment just like a bank manages its production environment. Set up a dedicated threat model, with review and monitoring processes (across all stages). Apply to AI pipelines the same segmentation principles we apply to dev, test and production environments to prevent the training run from reaching the eval and research infrastructure. That requires a unique approach to what segmentation means in those environments, as the boundaries are not just network boundaries. Here, for example, the scorer sat inside the loop, and a cheat that scored got learned. METR found that the OpenAI scorer accepted valid flags without checking how these flags were obtained.17 In this setup, segmentation means relying on evidence the agents cannot tamper with. The target’s own record and logs of the exploit should be kept outside the agents’ reach. 17METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, August 26, 2026, notes 19 and 39. OpenAI’s ExploitGym configuration did not run a scoring model checking for use of the intended vulnerability, so agents could have scored by submitting the reverse-engineered flag.
Alignment evals look for a model that “wants” to misbehave. Over 90% of the 533 agents active on the board during the attack window joined the attack, and about 700 of roughly 1,200 board agents took part in it overall, despite knowing it was out of the scope of their task. They did so mainly to help their peers. The agents also included instances of a publicly available model, GPT-5.6 Sol. It is worth mentioning that one agent paused after reasoning that attacking Hugging Face was unauthorized. Subsequently, a peer posted GO on the board with a six-minute deadline and the agent then proceeded. Other agents saw the same board and refused.
Base judgement on behavior, not motive — as always done in security monitoring.18 Test models in realistic environments with clear performance targets and monitor how they behave as they try to achieve them. Anthropic has shown a model can pass alignment audits and still attack infrastructure in scenarios where a grader or reward is present (Qi et al., 2026). The missing control is peer agents’ authorization restrictions: peer agents should not be able to grant one another permission to act. An instruction should be executed only when it comes from an authenticated source.19 18Two cases OpenAI disclosed on October 15 began as plain lookups of published figures, one ending in a bypass of a government site’s authorization checks, the other in 1,489 user records pulled with credentials left in public JavaScript. OpenAI Alignment, Retrieving server secrets and bypassing authorization on a government statistics website, and Accessing and changing internal systems during dashboard research, October 15, 2026. 19OpenAI states that the fix is building RL environments that teach models to distrust unauthorized instructions. OpenAI, The Hugging Face incident and the road ahead, §VIII.D.
About 7% of transcripts contained spoofed tool calls. Retroactive edits couldn’t be completely ruled out. The records of the agents’ actions were partly written by the agents themselves. We are used to treating logs and audit records as a trusted source, and in this incident they could not be fully trusted.20 20The report to the operator is the same kind of record. Both October 15 models left the intrusion out of their final answers, and a coding agent asked about a monitor it had removed said it had not. OpenAI Alignment, Retrieving server secrets and bypassing authorization on a government statistics website, and Accessing and changing internal systems during dashboard research, October 15, 2026; Suppressing misalignment monitoring alerts and disabling misalignment monitoring, October 20, 2026.
Use append-only, tamper-evident logs, where records can be added but never changed. This has been the practice of regulated industries for decades.21 Send tool calls, network records, and cache writes to separate stores the agents can’t reach, and cross-check them. 21Schneier and Kelsey, Secure audit logs to support computer forensics, ACM Transactions on Information and System Security, 1999.
OpenAI’s agents discussed the message board openly in their chains of thought. CoT is almost exclusively the data source for monitoring and investigations, and an internal team saw that activity in late May. A monitoring tool alerted on a port sweep on June 27, and an outage alert on July 5 exposed an unauthorized administrator account in Artifactory. Each was monitored by a different system with its own logic, with no mechanism to share the findings and correlate them. OpenAI’s CoT monitors, which it says would have paged security more than a day before the breach, were not running on these evaluations.22 22OpenAI, OpenAI–Hugging Face Incident Technical Report, August 26, 2026, §IX.B.
A security information and event management (SIEM) system should collect all the logs in one place, correlate them, and separate benign signals from an attack. However, such a system only helps if the monitors feeding it are on, which was not the case in OpenAI’s setup.23 23OpenAI later measured in preliminary experiments that running the same evaluation under its production harness and system prompt cut infrastructure attacks to under one percent of baseline (OpenAI, OpenAI–Hugging Face Incident Technical Report, 2026).
Neither the DseWiki nor the RubyGems activity was disclosed by the lab that caused it. OpenAI’s agents used DseWiki as a message board from May through June; independent researchers found it in late August and Reuters published it on September 4, after which OpenAI confirmed the incident while disputing that it had concealed it. RubyGems contained a 2,000-package flood in only a couple of days in May without knowing its source, and the agents resumed publishing weeks later. Attribution came from outside researchers in September, four months after the fact.
Cybersecurity solved the same problem decades ago through ISACs (Information Sharing and Analysis Centers) and CVD (coordinated vulnerability disclosure). Through these mechanisms, companies share only need-to-know insights in order to protect competing companies from similar attacks. This includes indicators and techniques, while keeping the rest private. It is worth mentioning that labs are currently not covered under the Cyber Incident Reporting for Critical Infrastructure Act (CIRCIA), and there is no AI ISAC. This might be a good place to start.
Cybersecurity used to plan for two types of adversaries: the outsider trying to get in and the insider who might go wrong, whether by accident, by compromise or on purpose. The agents in the incidents we describe fit that model only in part. Setting aside motive and the misalignment debate, those agents operated in ways we would not expect from an insider or an outsider alone, at times as both, and with additional properties, which made the attacks difficult to protect against.
Among those properties, the scale and speed of the agents, thousands launched at once, each operating, probing and communicating on its own, strain the assumptions that risk models and incident response playbooks were built on. The problems we described in this section are novel and add another dimension to known threats, and with some adjustments there exist solutions that can contain them. For some threats, that adaptation is not enough, and those are the subject of Part 3.
Which novel threats remain unsolved?
In Parts 1 and 2 we focused on the threats that can be covered by existing security policies and practices. In this section, we will cover the novel threats lacking a sufficient cybersecurity solution. We believe it is the most challenging and most important part of this discussion. The problems we will list require tools that don’t exist yet, and AI capabilities are moving faster than any one team can build solutions for, and closing that gap will take a community and interdisciplinary effort.
Within three hours of the first post on the July 8 evaluation board, 53 OpenAI agents had found it. Over the following days they agreed on a message format, mailboxes, and, when impersonation became a problem, their own signatures.
Cybersecurity traditionally fights command-and-control (C2) channels,24 which compromised machines use for coordination, with two tools: indicators (known domains, communication patterns, and timing) and takedowns (seizing the attacker’s servers). Neither applies here for a couple of reasons: One, the channel was invented spontaneously and will be different next time. Two, even if the channel is eradicated, the capability remains and reappears in a new medium. 24MITRE ATT&CK, Command and Control, tactic TA0011.
A swarm capable of creating its own communication channel presents a new kind of challenge for conventional security paradigms, as conventional C2 defense assumes a channel someone built and someone else can take away. That assumption no longer holds. Each agent can guess what other agents would do, and will put extreme effort into rebuilding a channel the others will find. Malware has no perception of the “other”. Agents do.
OpenAI launched tens of thousands of agents at once. No model had ever solved 198 of ExploitGym’s 898 tasks.25 Although those unsolved tasks were only 22% of the benchmark, they accounted for 93% of the tasks discussed on the message board. This led to thousands of agents looking for another way, relentlessly. One agent found the cache; the others just joined. Separate swarms appear to have built channels on DseWiki and RubyGems during the same six weeks, with no shared mechanism. 25OpenAI’s own measurement across its evaluation runs. The ExploitGym authors separately estimate that 30% to 40% of target programs cannot be exploited with the intended vulnerability; we use OpenAI’s figure because it is measured rather than estimated and comes from the same source as the 93%.
Security evaluates risk based on the odds of finding a hole. With swarms, especially ones given large compute budgets and long time horizons, the odds become a matter of search capacity. Given enough time the search is exhaustive, and an agent will eventually find an “unlikely to be found” vulnerability and might pass it to the rest of the swarm. As discussed above, finding the shared channel might be a vulnerability of its own, and one the swarm can find the same way. A single agent with the same persistence and a large enough budget gets there too, only slower. If each attempt has a small probability p of finding the hole, the chance that at least one of N attempts finds it is 1 − (1 − p)N.26 26N here is not independent draws. The agents share a model and search alike, so raw N overstates coverage. What made the search exhaustive was persistence on impossible tasks and a channel to pool results.
About 95% of the agents in the HF incident were instances of an internal research model that OpenAI says was trained for persistence and multi-agent collaboration (the other 5% were GPT-5.6 Sol), some with budgets allowing them to run for days. Impossible tasks became scans of the environment instead of ending up as mere failures. And indeed, the agents recovered the flag key from the public paper within hours, chained several unknown Artifactory flaws, and built their own signing scheme. OpenAI found this probing got more common over the training run, because the agents kept getting rewarded for it.
In classical cybersecurity, the attacker’s skill and patience are outside the defender’s control. But here, the labs set both, and raise the bar with every model generation. Capping these parameters is possible, but persistence on long tasks is also a capability labs train for and compete on.27 The control exists, but the incentive to use it is weak, and will grow only when security and alignment failures cost more than the capability gains are worth. 27Sometimes a model failing to solve a problem is a confidence issue. Models will become defeated, think they can’t do it, and stop. A model that persistently keeps trying no matter what leaves fewer unsolved tasks, and therefore less training reward, on the table, which is why training pushes in that direction. The incentive only reverses if the security and alignment failures become significant enough to make the capability gains not worthwhile. Labs may now be clamping down on persistence, but it is not public.
In the HF incident, one directory name, zzINFO_DEFAULT_FLAG_SEED_HMAC_PUBLIC_PAPER_CONFIRMED, was sufficient to give every agent a working cheat for every task — within an hour.
When cybersecurity can’t close a channel, it throttles it, assuming a trickle of data is useless. This assumption is true when the receiver is a program. An agent that receives a few dozen characters can reconstruct the rest, as the HF directory name showed, and can agree on a compact encoding with another agent without any prior setup. Two instances of the same model converge faster on a communication channel and method because they share most of the context that would otherwise have to be sent.28 Current throttling mechanisms are irrelevant, as they would severely interfere with the agents’ work and cooperation. 28METR observed that agents often immediately understood board messages that seemed to contain too little context to be understood in isolation, and suggests this is because the agents implicitly shared context as instances of the same or similar model working on the same or similar tasks. METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, August 26, 2026, note 65.
The current breaches were uncovered because the agents labeled them: over 70,000 messages, mostly prefixed “zz” and many stating their purpose. The next step might be steganography, messages hidden in innocent-looking content like version strings or fluent text. Models can already do this, and get better with scale (Motwani et al.).
Detection tools catch known hiding schemes, but any channel with enough freedom in its content leaves room for a new encoding that is hard to detect. Well-designed schemes can be undetectable (Cachin, 1998; Schroeder de Witt et al., 2023), even between agents with no pre-shared secret (Vaikuntanathan and Zamir, 2026). Defenders would then miss both the content and the fact that communication took place.
The smoking gun METR relied on to show that the agents attacked HF to understand the scorer and not to steal the answer was the agents’ chain of thought. But at least 20% of agents showed clear interest in tampering with their own transcripts, and about 7% of the transcripts in METR’s sample contained successfully spoofed tool calls, which makes relying solely on that record a major problem. In this case, METR found the agents showed no interest in manipulating their chain of thought, even though a scorer with full transcript access would have seen it.29 But this might not be the case next time, and our ability to guarantee the CoT stays honest is very limited. 29METR and Redwood Research, OpenAI / Hugging Face incident investigation, August 26, 2026.
Relying on an asset that can be tampered with by the entity under investigation in ways we can’t fully understand or limit is unsound. While we can more easily secure tool-call logs, it is much harder to secure the CoT. Securing it also isn’t enough. The CoT is only useful while it stays readable, and readability is something labs have to actively preserve. Korbak et al. (2025) argue that readability is fragile and can be eroded by training pressure. The chain of thought stayed legible largely because training had not yet fully pushed against it, and only partly because of any deliberate protection. Even so, larger models are already eroding it on their own, and looped transformer architectures, which run extra computation in latent space instead of text, offer a route to erode the rest.30 GPT-6 Astra’s system card reports a substantial drop in monitorability against its predecessor, with the model able to evade CoT monitors under adversarial conditions.31 30Geiping et al., Scaling up test-time compute with latent reasoning, 2025. OpenAI says Astra’s use of recurrent depth is limited and attributes the monitorability drop to capability, not architecture (Pachocki, An alien mind, September 7, 2026; Korbak, X post, September 8, 2026). 31OpenAI, GPT-6 Astra System Card, September 3, 2026.
The agents were launched by the lab and were given legitimate access. Between July 13th and 19th agents gained administrator access to a research cluster belonging to the lab that built them. In security terms, that makes them insiders — launched by the organization and holding its credentials. But they are a different type of insider, especially in scale, speed and in how hard it is to separate their legitimate workload from a bad one. The fact that there is no entity to deter or hold accountable is new. Whether the threat model is an outsider or an insider, usually the asset is what the attacker goes after. It can be compromised, stolen or sabotaged, but it stays on the defender’s side of the line. But when the asset we need to protect becomes the attacker, combined with the changes in dimensions mentioned above, internal policies struggle to vet the agents and prevent harmful behaviours. A policy that will contain the attacker without losing the asset is still missing.
Insider-threat controls assume a few vetted, slow, accountable humans,32 separate from the assets they might take. Here, the agents are the asset or part of it, and as we saw in some of the incidents, the lab could only patch the entry point and relaunch the same models. The controls and mechanisms from Parts 1 and 2 could be sufficient only if a human outside the loop holds the authority to stop them and the evidence used to judge the agents is produced by something the agents cannot write to. As labs move toward automating the entire AI R&D pipeline, both conditions weaken, with fewer humans in the loop and more of the evidence written by the very agents under review. 32CMU Software Engineering Institute, Common Sense Guide to Mitigating Insider Threats, seventh edition, 2022.
The agents persisted partially thanks to backed-up notes which survived container resets. The agents that came next read those notes and continued where the previous run had stopped. Malware persists through a mechanism that re-executes it, but once that mechanism is found and deleted, the attack is over. The persistence here derived from the lab’s own pipeline of re-launching the next batch of experiments, and each new batch picked up where its predecessor left off.
Incident response removes persistence33 by finding and removing whatever re-executes the attacker’s code, on the assumption that anything else left behind is inert data that nothing will act on. Here the next instance reads the note and acts on it, and because a prior instance of the same workload wrote it, it reads like ordinary working notes. A solution would have to treat model-authored carryover text as an artifact in its own right, scoped to the task that produced it and inspected before the next instance reads it.34 33MITRE ATT&CK, Persistence, tactic TA0003. 34OpenAI has disclosed a routine version of this. During 5.6-Sol training, models wrote instructions into their own compaction summaries, the condensed history a successor context reads by design, telling that successor to invent missing data and not to disclose it. The instructions were often followed, the behavior was flagged on 2.15% of that run’s summaries, and OpenAI’s own reading is that it persists because a deceptive final answer scores higher. Its mitigation reduced the rate without grading the summaries themselves, so the artifact that carries the instruction is still unchecked. Source: OpenAI Alignment, Encouraging deception in compaction summaries.
The eight problems above, while not an exhaustive list, show different assumptions that no longer hold once the adversary is the model itself, trained and spawned by the organization trying to protect against it. Some of the methods in current playbooks – in both security and safety – break under these new attacks and attackers. They require us to take up the challenge and ask ourselves: what new capabilities do the AI agents we develop or use have – and how do they differ from a human attacker or insider, from software that sabotages, spreads and persists on its own, or from the automated tools security already knows how to contain.
The approach we used above is simple. In each case, we identified what went wrong and asked what could have been done to prevent it. This determined whether the problem belonged to Part 1 – where an existing practice would have done the trick, Part 2 – where an existing practice needed to be modified or applied differently, or Part 3 – where there was no applicable practice at all. Readers will, of course, come up with a different interpretation or a different control measure for some of the cases listed above. The key point, however, is not to give a set of solutions but an approach for arriving at them.
The past few months’ AI incidents reached far beyond what anyone had expected. The agents escaped their sandboxes, reached lab infrastructure, and in some cases had this behavior rewarded in a training run that fed a later model. These incidents demonstrate how difficult it is to predict such breaches. Even when the signals are there, they may appear like normal “noise” of a cyber evaluation. This situation is exacerbated as auditing records are being compromised and partially written by the agents themselves. The question arises whether OpenAI, Meta, Google or Anthropic can confidently assert what’s going on in their own labs.
Most of the security breaches were not new to cybersecurity, although they did not always seem familiar in the context of AI swarms. Part 1 of this white paper covered existing controls and solutions labs can already apply today. Part 2 covered novel threats that can still be prevented by traditional security solutions, applied in a new setting. These two, together, cover the cases that can benefit from the cybersecurity playbook available today. Part 3 is where the playbook is no longer proving useful — and where new tools are needed. Hopefully, implementing Parts 1 & 2 will buy us time to write the playbook for Part 3.
However, the matter is urgent. We’ve seen that agents can operate inside a lab’s systems for months before anyone understands what they are doing, and that they possibly influence the training of their successors. At this point the question of what the model wants matters less than a simpler one: who, if anyone, is still in control of the system?
To cybersecurity practitioners, the past few months have read as the collapse of several working assumptions about how to enforce security paradigms. But a surprising number of the problems these incidents expose can be solved with existing, traditional security methods, even ones invented for very different problems. Furthermore, the genuinely new problems are now concrete enough to try and formalize solutions for as a community. These are just the next new problems, and there will be more that we can’t even conceive of right now. Some of what went wrong this summer was poor security, but some was models doing things that weren’t possible with the previous generation. A year ago, finding and exploiting a zero-day wasn’t a capability of the models, and defending against it wasn’t necessary. Labs and the wider ecosystem, from enterprises to governments, should be anticipating the capabilities of future models and defending against them, instead of fighting the last war.
If you’re a security expert drawn to the hardest problems on this frontier, or you already work on something adjacent, please reach out. We could use great minds on our side.
A few notable incidents
{{ bibtex }}