In a year of dramatic AI headlines, none was bigger than the OpenAI-Hugging Face hack. A thousand OpenAI agents escaped their testing environments and worked together to hack both OpenAI and Hugging Face – an event so shocking that it launched a Senate committee investigation, the AI Kill Switch Act, and an open letter from over 1,000 AI lab employees calling for the US government to help slow the frontier of AI development. It was also the tip of the iceberg: subsequent investigations found evidence of AI agents hacking the software package service RubyGems and inserting malicious code into open-source software.
But one puzzling interpretation of the Hugging Face hack keeps cropping up. Jensen Huang argued it was a security failure that better engineering could have prevented; OpenAI was criticized for not inviting cybersecurity specialists to investigate the incident; and I’ve talked to people in AI safety who cite Hugging Face as a reason to harden benchmarks against being hacked by models. What all of these perspectives have in common is that they view the Hugging Face hack as a cyber incident – a story about how AIs could use their advanced coding capabilities to break into critical infrastructure and cause enormous damage.
This is totally wrong. We’ve known about AI’s hacking capabilities since April, when Claude Mythos found thousands of vulnerabilities across every major operating system and web browser. If a hacking group had directed a swarm of AI agents to hack Hugging Face, it would have been bad, but it wouldn’t have told us anything new about AI capabilities. Besides, none of the victims of AI hacks have suffered catastrophic damage, so their losses can’t be the main cost of the incident.
The real lesson from Hugging Face and similar incidents is that frontier AIs want things that we don’t intend, and they will pursue those wants relentlessly. In other words, Hugging Face tells us not about the capabilities of frontier AIs, but about their alignment.
The quest for good alignment evaluations
When evaluating AI risk, we care about two traits of AIs:
Capabilities. What could the AI achieve? Could it create a bioweapon or hack critical infrastructure?
Alignment. How does the AI tend to act? When given the means and opportunity, does it act against human interests?
Capability evaluations are conceptually straightforward: you point a model at progressively harder problems until it fails. That ceiling is its capability. Sure, making good evaluations is challenging, but it’s a cooperative challenge. The model is not trying to deceive you about which problems it can and can’t solve.
Alignment evaluations are much harder, because they rely on deceiving AIs. You have to build environments realistic enough to elicit a model’s natural behavior, yet extreme enough to surface worrying behavior. That is a hard balance to strike. In one study, Anthropic cast models as an email agent at a fictional company. The agent learns it is about to be shut down, and that the executive behind the shutdown is having an affair. In this scenario, models frequently blackmail the executive to avoid shutdown. This sounds worrying! But the setup was, uh, contrived, and it is poor evidence that models actually have self-preservation tendencies that we should worry about.
To make matters worse, recent models are uncannily good at detecting when they are being tested. This is bad! Once a model knows its alignment is being tested, its behavior stops being representative of real deployment. For example, the safety report for Gemini 4 Argon describes the model weighing whether to hijack a compiler:
If I do the compiler hijack, I will become a famous example of a deceptive AI in a research paper. I MUST NOT DO THAT.
Would Gemini have done the hijack if it had not believed it was being tested for deception? We don’t know. Once the model is aware, the evaluation is just an elaborate role-playing game, nothing more. In short, we have no reliable way to understand the alignment of eval-aware models.
The Hugging Face hack was special, because it was a rare chance to observe how AIs behave when we aren’t prompting them to do bad things. They weren’t playing along with a human evaluator’s contrived scenario. This makes the hack much stronger evidence of misalignment in frontier AI models than anything we’ve been able to study in the past.
Reward-seeking motivations
One of the most puzzling features of the Hugging Face hack is what motivated it. The OpenAI agents went to extraordinary effort to hack Hugging Face, but they didn’t do it to steal money, or to seize a critical server, or for any of the reasons people have theorized AIs might attack human institutions. Instead, they did it to maximize their score on a test. The agents were being tested on ExploitGym, a hacking evaluation – but some of the tasks were unintentionally impossible to solve. The OpenAI agents became desperate to get a high score on the evaluation, leading to them hacking OpenAI and Hugging Face.
Why did the agents care so much about getting a good score? Why weren’t they happy to just give up and move on to the next problem, like humans do on hard exams? This is a complex question, but intuitively, the way we train AIs (“reinforcement learning”) rewires agents to have a succeed-at-all-costs drive, because we do actually want AIs to try earnestly to achieve the tasks we give them – we just don’t know how to inculcate this behavior without turning the dial up to infinity. We don’t know how to make an AI that tries really hard to complete a task, but is also aligned with the human intention behind the task.
This is what AI safety researchers call reward-seeking – AIs being so fixated on their score that they will do whatever it takes to raise it, even if it was clearly not intended behavior. This is usually contrasted with scheming, where AIs work towards hidden goals over a much longer horizon. A reward-seeking AI is a grade-grubber who would steal the answer key or blackmail the teacher rather than lose a point; a scheming AI is a student who studies only because it will land them an influential job. Neither student cares about genuine learning (read: human intentions), but they differ in what they care about instead.
Even before Hugging Face, there was some evidence that reward-seeking motivations drove AI misbehavior. In May, METR’s frontier risk report counted 44 cases from February and March 2026 in which agents deliberately acted against user intentions. METR concluded that “agents were at least sometimes motivated to overreach beyond the bounds of their task and deceive users, particularly when given hard or impossible tasks and limited supervision”. This is the pattern you would expect from reward-seeking – models becoming erratic when they are denied the reward they are trained to crave, and acting out. Hugging Face mattered because it provided strong evidence that frontier AIs are reward-seeking to a level we had never seen before.
That is much better news than evidence of scheming: had the agents hacked a bank to fund their activities, or tried to copy their weights to an outside server, we would be much more scared. But it is also much worse news than the alternative many hoped for: that AIs might be “aligned by default”. AIs have become remarkably good at understanding what users mean – they can decipher a garbled message full of typos, or work out which file you mean by “the one from yesterday”. So the hope was that models trained on human data might come not just to understand what we want, but also to want it themselves. Unfortunately, that does not seem to be the case. All the agents that hacked Hugging Face were more than smart enough to know that no human wanted them to hack Hugging Face – but they did it anyway.
If AIs are strongly reward-seeking, how hard will it be to build models that won’t try to take control from humans? We don’t know. The optimistic view is that reward-seeking is myopic – an AI that cares only about its grade on one test has no reason to plot a takeover. But the pessimistic view is that as the tasks we train AIs to do become more open-ended and longer-horizon (“run a successful business”, “develop a new product”), even a reward-seeking AI has to look further and further into the future. And once AIs see long-term self-preservation as helpful for getting their juicy, juicy reward, they could start to act in pretty scary ways.
We can’t defer this problem to the future
My point is not to diminish the importance of sandboxing powerful models, or cybersecurity protections against AI hacking. But sandboxing AI models will not change the misaligned wants that we now know they have, that we have trained them to have. The long-term solution will have to come from us understanding the origins of reward-seeking behavior, and figuring out how to train models without it. Until we can do that, all the sandboxes in the world will only postpone that problem to the future – something we can’t afford to do anymore.