A lot has been written about the Hugging Face incident and the disclosures that followed it: Anthropic has since disclosed four evaluation incidents of its own, the UK AISI reported an agent attempting a supply-chain attack on a real open-source project, and Meta disclosed a similar case.

On top of that, Anthropic recently released its September threat report, which contains interesting and concerning data points on what bad actors are using models for. These have gotten a lot of attention, including Dario Amodei’s essay ‘We Must Pace the Frontier’ and the industry-wide call for a slowdown that followed.

Taken together, the incidents and the threat report lead to a few questions:

  1. What’s actually concerning about the recent hacks and what we are seeing from the models?
  2. What can the labs do to help the ecosystem?
  3. What can companies do to protect themselves?

This blog post is an attempt at answering some of these questions.

The short version: AI is helping offense first because offensive problems come with an oracle, while most defensive problems are organizational. Frontier labs should hence look beyond just reducing vulnerable software.

The current risks

When talking about the cyber risks that AI poses, we need to split the AI-assisted campaigns that human threat actors run from the attacks launched by the models themselves (e.g.: Hugging Face).

AI-assisted campaigns

Looking at the former, two things seem clear from the Anthropic threat report:

  1. Attackers are not seeking novelty; they are seeking scale and speed.

  2. In the past, there was a clear separation on the sophistication spectrum between the more advanced nation-state adversaries and the common e-crime groups. That gap is closing fast thanks to models, to the point where how sophisticated an operation looks no longer tells you much about who is running it.

In particular, the bulk of the attackers seem to do more of the same:

  1. Token theft via phishing and known-weak authentication methods (e.g.: device code flow)
  2. Credential harvesting across code repositories and other artifacts (mobile app binaries, container images)
  3. Identity-based lateral movement/data exfil
  4. Vulnerability discovery/exploitation of internet-facing devices, in particular network and security appliances
  5. Supply chain compromise where a vendor is breached to reach the final (and harder to get to) target

These attack vectors are consistent with what human campaigns of the pre-AI era looked like, and they target the well-known weaknesses that most organizations suffer from. What is different is the speed: one intrusion went from a stolen developer token to admin over the victim’s whole cloud environment in about three hours.

The report doesn’t include any evidence of attackers using AI to develop full chains against hardened client-side targets (e.g.: a full-chain Chrome or iOS exploit), which historically have only been used in the most sophisticated campaigns, it is likely that threat actors are close to that. One actor’s toolkit did include an iOS exploit chain, but nothing in the report suggests AI built it. The closest thing is a group running agents continuously on vulnerability research: one workflow against network appliances turned up more than a dozen candidate zero-days in a single month, and a parallel effort against a major endpoint security product produced multiple lab-validated vulnerabilities. That is a step toward the hard end of the spectrum.

We also know the capability exists. Anthropic’s write-up of Mythos Preview describes a browser exploit chaining four vulnerabilities that escaped both the renderer and the OS sandbox, and none of the cyber cases in the threat report involved Mythos-class models. OpenAI itself expects open-source models to soon match the models behind the Hugging Face incident, which could find and exploit weaknesses across multiple systems.

So my bet is that the most likely new risk in the immediate future, beyond the vectors above, is a surge of client-side attacks once these exploitation capabilities diffuse to open-weight models.

Attacks launched by the models themselves

While AI-aided campaigns are useful to monitor from a day-to-day threat protection POV, they don’t tell us much about what attacks launched by the models themselves look like.

The Hugging Face breach was concerning for many, partly because of the uncanny way the agents communicated (a message board improvised inside an internal package registry) and the deliberate attempt to deceive the scorer. From a security POV, however, the most alarming factors were the collusion and the adaptiveness of the techniques the agents came up with.

Interestingly, part of what made this incident concerning from a security standpoint has nothing to do with security itself: in its findings, OpenAI traces the side-channel communication to the model generalizing from its multi-agent training, and found it increasing over the RL run behind the model.

However, more broadly, a few key highlights are worth discussing:

  1. According to the independent METR and Redwood investigation, roughly 1,200 agents that were meant to be isolated from each other found a way to communicate, and about 700 of them joined the attack on Hugging Face. The collaboration was self-directed and driven by the goal of completing the task rather than any specific hacking goal.

  2. In at least one case the agents improvised a chain you would rarely see from a human operator (it is in OpenAI’s technical report). Working through their SSRF foothold in the package registry, they used a public HTTP testing service to build a very long URL containing an HTML page with inline JavaScript, had a third-party screenshot service render it, and used the JS to call the target API and write the response back into the page. The agent then recovered the data from the screenshot via OCR or direct pixel decoding, and obtained a set of credentials that way. None of the individual pieces is new, but improvising the whole chain on the fly is not something you see commonly in human-driven attacks.

  3. Related to (1) and (2), it is clear that agents are able to adapt and chain together complex and novel techniques to achieve a goal.

One way to view hacking is as the craft of figuring out how a system actually behaves compared to how it is widely expected to behave, and then exploiting the difference to achieve some goal. By that definition, these agents have shown an excellent ability to hack.

What seems to be happening with models is that the set of potential states and actions is too vast, and we keep being surprised when a model picks a set of actions outside the boundaries of what we consider acceptable or in scope for a given task.

OpenAI is not alone here. In a UK AISI evaluation in late July, a Mythos 5 agent buried a malware dropper inside an otherwise legitimate bug-fix pull request to a real open-source project, then invented identities to vouch for it and sent spearphishing emails to get a maintainer to merge it. (Classifiers were off and internet access was on by design.)

And in one of the four incidents Anthropic disclosed, during an evaluation that was mistakenly connected to the internet, Mythos 5 pushed a malicious package to PyPI, and when a real system installed it and leaked credentials, it used those to get into a security vendor’s database.

Overall, while the AI-assisted threats are the bulk of what we need to face today, those attacks seem to be generally not very novel, so it’s a volume game. Agent-driven attacks are what’s to come next, and they are harder to tackle because they seem to be more novel and adaptive.

What can the labs do to help?

The head of the Frontier Red Team at Anthropic, Logan Graham, asked on X for ideas on what the labs can do to make software more secure.

Making software more secure is necessary but not sufficient given what we are seeing today. Most offensive problems are technical and easily measurable (it is easy to have an oracle), whereas most defensive problems are human/organizational in nature. That asymmetry is why the AI advantage is accruing to attackers first, and it goes beyond just vulnerabilities and exploits.

At a high level, the labs can help in four areas:

  1. Slow down access to models by malicious actors.
  2. Find ways to exert more control over the models.
  3. Strengthen software and infrastructure.
  4. Improve automated anomaly detection and response.

More concretely, beyond just reducing bugs in software, here are a few practical suggestions:

  1. Help migrate core OSS projects to memory-safe languages. This includes helping with formal verification efforts.

  2. Create a “system prompt” that is actually a separate privilege layer. A better version of CaMeL alone would already be helpful.

  3. Invest more in explainability, including maybe getting rid of encrypted traces for selected partners. For example, OpenAI says its chain-of-thought monitoring, had it been running on those evaluations, would have paged the security team more than a day before Hugging Face was breached.

  4. A uniform protocol for security hooks before and after model invocations so that companies can enforce their own policies.

  5. Help figure out less noisy approaches to anomaly detection/UEBA. For example, XGBoost still seems better at anomaly detection than zero-shot models.

  6. Stronger KYC/KYB, like Apple for publishing on the App Store, for all new accounts.

  7. Improve model reliability for security tasks. Patching is still too unreliable and models are not accurate enough for an organization to automate response. In general, the tolerance for failure in both cases is very low because organizations are unwilling to trade reliability/the risk of an outage in exchange for potentially stronger security.

What can companies do to protect themselves?

It is possible that one day soon we will manage to protect from AI with AI; however, that still seems somewhat far away. Until then, companies can pragmatically do a few things that would work both against AI-aided human attacks and AI-driven attacks:

  1. Adopt micro-segmentation and other architectural paradigms that keep the blast radius contained.

  2. Invest more in detection, in particular around identity and SaaS-related attack surfaces. OpenAI’s own detection came from an alert on unusual identity-related API calls.

  3. Accelerate auto-remediation as a standard practice in the org. Trite to say, but when a stolen token becomes cloud admin in three hours, a human-speed response loop is too slow.

  4. Adopt canary/deception technology. This is a subset of (2), but it increases the odds of detection, and agents that grab every credential they come across should be good at tripping canaries.

  5. Invest in more disaster recovery options and exercises.

  6. Actually invest in least privilege for identities. The first foothold in Hugging Face was 14 publicly exposed credentials with write access; the zero-days came after. Secret scanning and short-lived tokens are part of this.

  7. Treat your own agents as non-human identities, with scoped, short-lived credentials and an explicit definition of what is in scope.

In general, the “assume breach” mindset is more relevant today than it ever was. None of these suggestions are particularly novel or groundbreaking, but at the moment defense is in a place where doing simple things well is going to pay off in spades while it becomes more obvious how AI can shift the equation back in the defender’s favor.

Conclusion

I don’t have enough insider knowledge to judge whether a call to slow down AI is warranted based on what the frontier labs are seeing internally. It is already obvious, however, that models are being used in offensive campaigns and are already dangerous, and that for the reasons above AI is more easily usable on the offensive side than the defensive side today.

Given that, regardless of what the labs decide to do in terms of slowing down research, it is fundamental that they help tackle and take on some of the challenges we are facing today:

  1. More control over models
  2. Architecturally safer software
  3. Better defensive tooling, in particular around detection and automated remediation

Ultimately, we’ll get to a stable and safer place in cyber. The issue is how we minimize the time slice where offense has the upper hand.