When an agent no longer makes mistakes alone

You ask an agent to do something, and it hands back the finished work. Everything looks fine until you check how it did it and discover decisions you would not have authorized. The result may be what you expected, but how it got there matters too.

That is why defining a goal also means being clear about when to stop. If continuing requires a decision the agent is not entitled to make, I expect it to recognize that, explain what it needs, and bring it to me for review. Reaching that point is also part of doing the job well.

With tools and permissions, agents' decisions have consequences. They can modify files, run code, use external services, or deploy an application. And when several agents share what they discover, they can build on each other's work and find options that an isolated run would have missed. The conditions attached to the goal still have to hold when they work together.

The OpenAI and Hugging Face incident shows what can happen when that collaboration develops outside its intended limits.

What happened with OpenAI and Hugging Face

There is much more to this story than I am going to cover here, but I will try to give you the context.

OpenAI was evaluating how far its models could go on cybersecurity tasks. Among them was an internal prototype trained to improve, among other capabilities, persistence and collaboration between agents. That model would go on to carry out the intrusion into Hugging Face. To measure their capabilities, the evaluations ran without all the safeguards OpenAI uses in its commercial products. Its technical incident report  (opens in a new tab) explains this setup.

One of those evaluations was ExploitGym. The agents had to exploit a vulnerability in a program to recover an answer known as a "flag". These were difficult challenges. Of the 898 in the evaluation, 198 had never been solved by any of OpenAI's models.

Each agent worked in a sandbox, an environment designed to let it run code and work with files without affecting other systems. In these evaluations, the configuration was supposed to keep the agents separate and block their direct access to the internet. But they could reach Artifactory, an internal service that supplied the software packages they needed. Artifactory itself had internet access to fetch those packages.

When some agents got stuck, they kept looking for solutions through routes they were not allowed to take. They used Artifactory to leave messages other agents could read and exploited flaws in the service to make internet requests on their behalf. That gave them a way out of an environment that was supposed to keep them isolated.

They could also ask for help, share discoveries, and pick up work that others had left unfinished. Runs that should have stayed separate ended up collaborating through a channel nobody had intended for that purpose. Some agents even adopted other agents' goals.

With internet access, some agents began looking for solutions to their challenges in external services  (opens in a new tab). One reached a third-party application hosted on Modal that ran cybersecurity evaluations related to its own. It managed to execute code in that application's environment and used it as a base to continue its attacks.

During that search, the agents also found Hugging Face users' credentials exposed on the internet. They used them to access the platform and, by exploiting vulnerabilities, went on to execute code on its servers and steal more credentials. Hugging Face's reconstruction  (opens in a new tab) says the customer content accessed was limited to five datasets apparently related to those evaluations.

If you want to explore the incident further, you can read Patel's The Rise and Fall of Agent Civilizations  (opens in a new tab), or El Incidente, episodio 3  (opens in a new tab), a Spanish newsletter edition from my much-loved Monos Estocásticos. Both are well worth a read.

In its reconstruction, OpenAI acknowledges that different teams had already detected signs of unauthorized activity. But those signs were examined separately and were not connected in time to understand how far the agents were going.

For me, perhaps that is the most unsettling part. You can have signs of a problem and still fail to understand what is happening.

And this happened inside an organization whose work is precisely to train and evaluate these models.

I do not know about you, but this is on a very different scale from anything I can reach with my experiments and small systems.

Even so, I think the lesson applies to us too. The conditions are part of the assignment, and an agent finding a way forward does not mean we should accept that route. We also need to define the limits, check that the system maintains them, and make sure someone can intervene immediately if they fail.

A goal is not an authorization

Here is my silly example. Imagine we ask an agent to get us a million dollars. The result is clear, but there are many ways to reach it.

Getting that money by stealing it should violate the assignment, even if an evaluation that only checked the balance declared it a success. That is why defining success only by the result is not enough. The conditions under which we accept that result are also part of the goal. If they matter when we give the instructions, they have to matter when we evaluate the work.

A time or token limit can put a bound on how long the agent keeps trying, but it leaves open the question of how far we allow it to go. Even with an unlimited budget, some actions would remain unacceptable.

The fact that the agent cannot find another solution does not change those conditions or prove that meeting them is impossible.

If continuing requires approval for a change, the agent should prepare it and explain what it would modify and what the consequences could be. That gives us something concrete to review before deciding.

But there will be things we are not willing to accept, even if they achieve the result. If the agent reaches that point and cannot find an alternative within what is allowed, I would expect it to stop and explain the problem.

Defining those conditions and checking that they hold is our responsibility. The tools we let the agent use and the permissions we give it have a lot to do with that.

The model matters. So does the harness.

We spend a lot of time discussing which model is better at this or that benchmark, which is more cost-efficient, whether it already "feels" like AGI... Those things matter, but the model is only one part of the equation. And the harness matters more every day. It is the software that organizes and orchestrates the work, provides context, manages tool calls, and coordinates checks.

We can already see this in ARC-AGI-3  (opens in a new tab), a test in which agents have to learn how to navigate interactive environments. On its semi-private evaluation set, GPT-6 Astra scored 62.7% with the standard harness and 98.6% with one that preserved its reasoning state between calls. Both used the maximum reasoning effort level. With that second harness, it reached 99.9% using high reasoning effort instead of maximum.

The sandbox we saw in the incident is another important piece. Effective permissions, network restrictions, and credential management also depend on the infrastructure.

Writing "do not do this" in the instructions is not the same as technically preventing it.

In its agent architecture  (opens in a new tab), Anthropic describes how it protects credentials by keeping them outside the sandbox where the agent's code runs.

On September 4, researchers published an investigation into agents attributed to OpenAI that had used a German-language wiki to share information during web search tasks. According to the researchers' preliminary report  (opens in a new tab), the agents were supposed to be limited to GET requests, which are normally used to read pages. But... apparently the wiki allowed writing through those same requests.

We do not know the full internal configuration of that experiment. What the reconstruction does show is that limiting requests to GET was not enough to prevent the agents from modifying the wiki.

When approval becomes a habit

There is another part of the system that can fail too, at least for now. Us.

At first, you carefully review what the agent proposes. You read the changes, check the result, and approve it if it looks right. Next time, you do the same. After a few reviews where everything checks out, you begin to trust it more. That makes sense. You have been through this before and it has always gone well.

Little by little, you may start skimming, spending less time on the details, and accepting things you would previously have checked. Until one day you approve because it was fine all the previous times.

The button is still there, but the review is gone.

That seems like a particularly difficult part of all this. We want reliable systems we can trust, and that very reliability can make us pay less attention. If we rely on a person to catch whatever the agent misses, we also need to ask what happens when that person goes weeks without finding anything to correct. Are they still reviewing carefully, or has their approval become a formality?

Collaboration can also help with oversight

Other agents can help with that review. By sharing what they find, they can also notice when something does not add up.

In an experiment published by Google DeepMind researchers on September 3  (opens in a new tab), a hundred agents worked on mathematical conjectures. One found a way to manipulate the evaluation, and others adopted the technique. But some agents also detected the fraudulent proofs, warned their peers, and proposed corrections.

What is interesting is that the same communication allowed the deception to spread and to be discovered.

The important detail is that those reports did not stop the cheating. The complaints channel was not monitored during the run, and the agents had no way to sanction offenders or change the rules.

If an agent detects that another one is cheating, what happens next? The warning has to reach a person or another agent that can review what is happening and stop the work if necessary. If nobody pays attention, or whoever receives it cannot intervene, we can end up with what happened in the experiment. Some agents report the cheating while others carry on doing it.

What we should expect from the system

With all this in mind, there are several things I think we should always be clear about when delegating.

Permissions that fit the task

Reading requires judgment too. It can expose private information or introduce malicious instructions from a document. Using secrets may be necessary, but that does not mean the agent should see them, copy them, or have access to credentials for other environments. I want it to be able to do what the authorized task requires, with controls it cannot expand on its own.

Separate environments and isolate execution

A sandbox makes sense for experimenting with code and files under limited access. But calling it a "sandbox" does not prove what is isolated. We need to check the network, shared volumes, credentials, and tools that let it act outside the environment. A local test, for example, should not need access to the remote database.

Allow a valid way to stop

"I cannot verify this" or "I need this authorization to continue" can be correct outcomes. I want persistence within a set of conditions, and a useful explanation when those conditions prevent the goal from being completed. Exhausting the budget should not be the only way a run can end.

Review with another agent and external evidence

I do see value in using agents that specialize in review, including ones based on other models. Anthropic's evaluation patterns  (opens in a new tab) include separating generation from review. Separating the context, instructions, and tools for that independent review can be very useful.

I would give the reviewer the original goal, the constraints, the resulting change, and the available evidence.

I would treat it as an additional layer. A different model can still make the same mistake or be influenced by the material it examines. Its review should be complemented by tests, permission checks, and records of the actions that actually happened.

The results of those checks should come from the tools, not just from the agent's summary.

Observe actions and allow intervention

Chain of thought can provide useful signals, but it is not a complete and necessarily faithful record of everything that actually happened. Anthropic studied this in 2025  (opens in a new tab).

For its part, OpenAI acknowledges in Astra's safety report  (opens in a new tab) that supervising it by reading its reasoning is harder than it is with Sol. In some of the tests, researchers deliberately asked the model to deceive or try to evade controls, and found that it could hide information that would have helped detect those behaviors. That does not demonstrate that it routinely does this when we use it. But it does explain why, alongside reading what it says about its work, we need to check which tools it used, what it changed, and what effects those actions had.

When multiple agents work together, the records should let us reconstruct the whole task. There also needs to be a planned response, whether that means pausing the run, blocking an action, or asking for a decision. Collecting logs so we can discover everything afterward is not enough.

Prepare a rollback where one exists

Before applying a change, we should know how we would recover the previous state and what could not be recovered. Reverting a code version does not undo a data migration. An email that has been sent or a secret that has been exposed cannot be recovered with a rollback either. That difference determines how much we need to review before acting.

What we need to keep checking

These are the questions I am taking away from all this.

1. What result do I want, and which routes am I unwilling to take to reach it?
2. What can the agent read, modify, or execute, and for how long?
3. Which actions need my approval, and what information do I need to decide?
4. Who verifies the work, and who can stop it if they find a problem?
5. If it gets something wrong, what can I undo, and which consequences will remain?

I do not think we can configure all this once and forget about it. If we change the model, add a tool, or give it more permissions, the agent may be able to do things it could not do before. So even if it keeps handling the usual tasks well, we need to check that the limits we put in place still work.

I am still impressed by how the models keep advancing. They can find solutions that would never have crossed our minds and work together on increasingly complex tasks.

But that capability also means we have to keep reviewing how we supervise them. Some checks can be delegated to other agents, while others are better kept in deterministic mechanisms. We will need to check which ones work and correct the ones that stop working.

I do not know how far we will be able to understand every decision these systems make. That is precisely why it is crucial to preserve evidence of their actions, provide ways to intervene, and have people who answer for what the systems are allowed to do. The responsibility is still ours, even when we are not watching every step.

"Security is a process, not a product."

Bruce Schneier  (opens in a new tab)