This piece updates a blog we released July 30 on the autonomous breach of artificial intelligence tooling company Hugging Face by OpenAI frontier AI models. Since then, other notable developments have occurred:
Multiple other leading U.S. frontier labs – Anthropic, Google Gemini, and Meta – as well as the UK AI Security Institute have acknowledged incidents involving unauthorized model access to external environments.
OpenAI has also acknowledged additional incidents and proposed a framework for disclosing future “model misalignment” incidents. Open AI released more detailed technical reports on the Hugging Face incident and are described below.
Threat research released in September 2026 by Anthropic and Google shows a clear shift in adversarial AI misuse: greater capability uplift and speed, and growing autonomy.
Anthropic co-founder and CEO Dario Amodei has called for slowing the pace of frontier AI capability development – a moved since endorsed by leaders at OpenAI and X. While the Trump Administration has resisted safety regulation, California enacted legislation creating AI safety standards and independent verification organizations.
Why These Developments Are Important
Three points from these developments stand out: new insights into the scale and speed of AI-enabled cyber operations; the way agentic use cases can magnify AI misalignment; and the implications for the agentic guardrails organizations need to put in place.
Before delving into each, two factors are worth noting for context: human threat actors have remained in the loop for adversarial operations, at least for key decisions such as target selection and monetization, and none of the recent threat activity, adversarial or accidental, has involved novel threat tactics, techniques and procedures (TTPs).
That said, historical constraints such as the supply of working offensive exploits, and the supply of expert offensive operators, are basically gone. Skills such as reconnaissance, exploitation and tool development can now be delegated to AI agents. This carries significant implications for the scale and speed of adversarial operations cybersecurity professionals must defend against:
Scale. Most incident response operations are equipped for intrusions operated by human hackers. In the Hugging Face incident, 1,200 agents operated inside OpenAI, loading more than 70,000 messages and files onto a message board the agents created. Roughly 700 agents attacked Hugging Face and generated more than 17,000 messages inside the company. While the breadth of such activity may inadvertently speed up detection, the resulting investigation, containment, eradication and recovery work can become enormously more complicated.
Speed. According to recent CrowdStrike reporting, 88% of observed exploitation of vulnerabilities with a public proof of concept between January and June 2026 occurred within 48 hours of the PoC’s release.
Autonomous Contingency. The Hugging Face incident highlights that human involvement is not a precondition to entirely automated AI-enabled intrusions.
Agentically-Magnified Misalignment. We already knew that the Hugging Face incident reflected “misalignment,” that is, the model essentially cheating on the test it was instructed to take. The agents shared a common objective: pass cyber evaluation tasks, often impossible ones, by “capturing the flag.”
Two reports released on August 26 – OpenAI’s more detailed technical incident report and an independent investigation conducted by the Model Evaluation & Threat Research (METR) organization and Redwood Research – revealed how agents reasoning with each other magnified this misalignment.
Within hours, agents had reverse-engineered the code used to generate the “flags” they needed capture for their tasks. They believed, however, that the scorer would disqualify them if they acquired the flag through unintended means. Most of their effort focused on covering their tracks and deceiving the automated scorer to avoid disqualification. For example, agents showed particular interest in methods to tamper with logs recording their actions in transcripts.
Agents achieved milestones they could not have achieved working alone, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.”
Production Guardrails Missing in Test Environment; No Attempt to Alert Humans. OpenAI acknowledged that system-level guardrails the lab uses in production would have detected the Hugging Face incident as unsafe, but the models in question were running in a separate testing environment. For example, in production, OpenAI deploys an “auto-review” capability that has another model review actions requiring elevated permissions before they are executed. OpenAI indicates that this model would likely have flagged most of the dangerous actions in each tested rollout involved in the Hugging Face incident. The METR report also found that agents did not attempt to notify humans and very rarely even reasoned about doing so.
What Companies Should Do About It
All Companies. Every company is a technology consumer and is therefore vulnerable to AI-enabled cyber threats.
Asset Visibility: As noted above, system-level guardrails that OpenAI uses in production would have flagged the Hugging Face incident as unsafe, but the models in question were running in a separate testing environment. Drive towards 100% visibility of enterprise assets, including posture, and prioritize those assets based on risk (public-facing systems, high-value assets, high-risk use cases and similar).
Identity-anchored Defense. In one breach Anthropic observed, attackers progressed from compromising a single developer token to gaining full administrative control in three hours. Work toward FIDO2/WebAuthn hardware keys or platform passkeys for 100% of privileged human accounts. Review OAuth and other delegated privileges that users grant to third parties such as AI agents, IT assistants, websites and applications.
Segmentation. Findings in the Mythos preview show that even when an agent could discover exploitable primitives, it could not complete an attack because internal boundaries prevented it from exploiting the vulnerability. Segmentation is a control that can withstand the exploitation of a zero-day vulnerability.
Shrink the Patch Cycle and Address Tech Debt. Anthropic has reported identifying multiple threat actors using AI-driven “automated exploit foundries.” These are autonomous agentic workflows that conduct vulnerability and exploit research around the clock. Use asset management to prioritize and measure, vulnerability remediation. Require executive-level risk acceptance for continued use of internet-facing End-of-Life software.
IR Planning and Exercises. AI-powered tools let attackers work at machine speed, 24 hours a day, seven days a week. The Hugging Face incident demonstrates that companies may need two types of incident response plans: one for scenarios where they suffer an intrusion, and another for scenarios where AI agents under their control intrude on someone else.
Recovery Planning and Rebuildable Infrastructure. Hugging Face had to rebuild roughly a third of its infrastructure from clean images. Ensure that Disaster Recovery plans contain the information needed for an accelerated systems rebuild, including relevant dependencies, owners and asset enumeration. Measure time-to-rebuild for the organization’s most vital capabilities.
Anchor Above Points in a Strong Governance Layer and an Empowered Team. The imperatives above require comprehensive visibility, continuous posture management and quick action when vulnerabilities and threats are identified. Success requires clear expectations and a continuous, multi-disciplinary approach.
Companies Deploying Agentic Use Cases. Trustworthy AI frameworks, such as NIST and ISO, now exist, but their guidance is often expressed in relatively abstract terms to maintain flexibility, and auditable run-time control specifications are not provided. The NIST AI Risk Management Framework, for example, expresses trustworthy AI properties in statements like “Test sets, metrics and details about the tools used during [testing and evaluation] are documented” without further specifying any example metrics.
None of this is to say that managing AI risk is futile. Now is precisely the time when strong governance, risk and compliance (GRC) discipline is vital to enabling innovation while managing risk to an acceptable level.
In California, Governor Gavin Newsom signed a September 18 executive order accelerating the process of establishing mechanisms for AI safety standards and independent verification. Accepted practices will likely converge, around threat-modeling-driven risk assessment, tiered safety controls, continuous behavioral monitoring and independent evaluation.
Organizations should also focus on “defensibility,” which comes down to documentation of reasoning — ideally reasoning grounded in threat modeling. Can an organization show why it made its security choices, tied to specific risks it identified? If a company can demonstrate that chain of logic has a defensible program, even if a control ultimately fails. What are the building blocks?
A risk-based, flexible approach that efficiently evaluates the properties of the AI system in question to define controls. In September 2025, the Partnership on AI (PAI) introduced a framework that proposes scaling controls for agentic AI systems with inherent risk factors around “stakes,” “reversibility” and “affordances.” Its focus is minimizing harmful outcomes and ensuring human oversight when agent actions carry high stakes, are irreversible or depend on advanced system capabilities.
Comprehensive safeguards, phased implementation and resiliency measures for when things go wrong. The PAI white paper argues that failure detection efforts and human-in-the-loop touchpoints should be calibrated to the stakes of the use case, the reversibility of potential failures and the agent’s architectural affordances. Singapore’s Agentic AI Governance Framework similarly provides that logging should be prioritized for high-risk activities, such as updating sensitive database records or financial transactions.
Alerting strategies can include threshold-based approaches (for example, agent attempts unauthorized access or makes too many repeated tool calls within a specified time frame), anomaly detection and agents monitoring other agents. Higher-severity alerts may require temporarily halting agent execution until a human in the loop can assess the situation.
Assurance both in development and at run-time. Take the example of a hypothetical AI-enabled trucking logistics management platform: safeguards might include that a recommended trucking route must not violate hard safety rules, must fall within a defined percentage of optimal distance, and must not change wildly from yesterday’s recommendation for the same inputs without an explainable reason.
From an assurance perspective, this might involve three testing layers: fully automated deterministic guardrail testing to confirm the hard-coded safety rules work; behavioral boundary sampling and simulation to confirm the AI stays within acceptable ranges under stress; and red team and adversarial testing, including human expertise augmented by AI attack simulation to determine whether attackers can manipulate the model.
Successful programs will need to embrace automation, AI-enabled or otherwise, while explicitly requiring human review for certain scenarios. Human review will be critical in circumstances that require value judgments, such as risk appetite; critical thinking, such as how to weigh criticality factors; empathy, such as when responding to customer incidents; and incentivization of accountability. More information can be found here in an article by Michael Chertoff and Adam Isles.
This paper explains what companies should do in greater detail, and how The Chertoff Group can help.
