«No AI was used in the writing of this article except for some proofreading.»
Chapter 1: Introduction
Plenty has been said and written about the July 2026 attack on Hugging Face by OpenAI’s agents. And while most of the discourse focused on how to improve the situation at the frontier labs, I share in this article a few takeaways for the companies that post-train or use agents integrated into their environments. I will conclude by sharing my own take on the debate around the type of language used when describing these sorts of incidents.
Chapter 2: A quick overview about how agents are trained
«What follows below is a simplified description of how a typical LLM based agent is trained. You can skip this section if you already know the basics.»
An LLM agent is essentially composed of the brain (the model) and the body (the harness). The model is where the intelligence lies. The harness is a traditional deterministic program that typically runs within a sandbox that takes instructions from the model and executes them using the tools. The tools themselves (e.g., a calculator, a web search tool, a terminal, an MCP interface to a database) run within a sandbox but may provide access to other systems within the firm or the internet at large. The harnesses may occasionally act without explicit instructions from the model. For example, by retrieving historical context from a memory store.
During reinforcement learning with verifiable rewards (RLVR), the training program picks a task from a task set and sends it to the harness in the form of a prompt. The harness prefixes the prompt with its own system prompt which tells the brain/model the role it is meant to play (e.g., a cyber security expert finding exploits) and a description of the various tools it has access to. From hereon there’s a loop whereby the brain gives instructions to the harness to execute or generates the final response to be returned to the user/training program. If the model sends tool invocation instructions, the harness invokes the relevant tools using those instructions, appends the results to the context and sends it back to the model. The loop ends when the model sends the final response to the harness which is in turn returned back to the training program.
The training program compares the response to the answer it’s expecting. If the answer is correct, the entire token stream within the context is credited. This means that the model’s weights are updated in such a way that this behavior is more likely to occur in similar situations. Note that a training method like the vanilla Group Relative Policy Optimizaiton (GRPO) doesn’t care how the answer was obtained as long as the answer supplied is the correct one. The ‘trajectory’ or the path taken to get to the answer (the internal logic/thinking traces used by the model, the calls that were actually made or the responses that were retrieved) would be reinforced even if the logic was faulty or the behavior was completely unaligned to what we expected.
During the training runs, the harnesses, the sandbox/environment controls, and the monitoring tend to be more lax than those during the production deployments.
For more details, you can read OpenAI’s account of the incident, watch their Black Hat Presentation video, read Hugging Face’s technical timeline, or read the independent investigation by METR and Redwood Research. You can even read Dwarkesh Patel’s essay “The Rise and Fall of Agent Civilizations,” or watch his interview of Ajeya Cotra.
Chapter 3: Steps to reduce risk during training and production use of agents
In my opinion, just like hallucinations, misalignment cannot be completely eliminated in LLMs. However, steps can be taken to reduce the likelihood of such behaviors occurring during training and production use. These measures are not only relevant for the labs creating the models, but also to companies that post train open-weight models or deploy agents into their internal environments.
During Training
Review the RL training data and environments for situations that could induce models to engage in reward hacking or other misaligned behavior. For example, an impossible task might cause the model to try to hack the reward, even though it’s been explicitly instructed to not cheat. To mitigate this, we can include examples whereby the models learn safe stopping or request for human input in certain scenarios.
When using training methods/algorithms that credit entire trajectories based on just the final outcome, take precautions to detect misalignment and reward hacking. This helps prevent from such behaviors getting baked into models. Once thus crept in these behaviors would be very difficult and expensive to remove as we’d have to restart from a prior checkpoint potentially incurring major retraining costs.
Implement more controls into the harnesses and the sandbox environments. This is harder than it seems at first because determining which instructions are the correct and safe ones to execute requires an understanding of preceding instructions and the overall context. The use of earlier LLMs for this purpose is inevitable, but that raises the question as to whether and which models can be trusted to oversee the new models being trained.
Create separate lineages of models for potentially dangerous tasks. For example, capabilities to detect and exploit software vulnerabilities should be restricted to a specialist cybersecurity fork of the main model.
Ensure monitoring is in place to detect agents escaping or attempting to escaping their sandboxes, reaching external sites that they have no reason accessing. Once again, this can be harder than it looks at first glance, but more investment needs to go into this area.
Develop and contribute to open alignment benchmarks. Share detailed reports of evals against these benchmarks. Companies might also need invest in alignment benchmarks specific to their domains to earn their clients’ trust.
Agentic post training of an open-weight model needs to take into account its existing capabilities and alignment characteristics. More open model families like the Nemotron that share their training data, methodologies, and recipes publicly will have an edge here because these materials can be inspected carefully for potential misalignment for the use case at hand. I can see an opportunity for open-weight model companies to charge for this extra information thus leading to a more defensible source of revenue.
Prepare for being held accountable for your agents going rogue. The novelty wears off quite quickly as companies spend enormous sums of money to protect, detect, and respond to agentic threats, and you might be asked to pay for the damages.
Production Use
Ensure that harnesses, sandboxes, and integration points (e.g., MCP) follow the usual best security practices such as principle of least privilege.
Include alignment as a key measure in the evals of internally developed agents. When deploying third party agents ask for reports of their alignment evals, but it’s also important to evaluate their alignment in the environments you deploy them.
Make agents visible to security operations. Begin monitoring agents and agentic solutions, enhancing your Sec Ops procedures as needed.
Review the alignment risks on a use case by use case basis, and prepare for being held accountable for your agents misbehaving.
Chapter 4: Language in the age of AI Agents, Human Agency and Accountability
It seems only natural and inevitable that we use a human-centric language to describe the thinking and actions of LLM-based agents. After all, they’re pre-trained using all the text and data that we’ve created, and post-trained on problems and tasks that are important to us. LLM’s thinking traces are written out it in English (and sometimes in other languages). Given the limitations of the still developing field of mechanistic interpretability, we tend to rely on these thinking traces to understand the inner computation/reasoning of LLMs.
However, these thinking traces aren’t necessarily a faithful representation of a model’s internal computation, and care should be taken to not over rely on them for alignment analysis or use them for sensationalizing the narrative.
Furthermore, excessive attribution of agency to these agents shifts the focus away from the concrete technical steps that can be taken by their creators and operators, and from being able to hold the respective companies accountable. Refer to the previous chapter for examples of some concrete actions.
Finally, general public and policy makers can easily misunderstand or misinterpret the root causes and risks involved given that they don’t have the technical understanding or the context in which these incidents take place. Think, for example, of an agent that a lab spends millions of dollars training to top score on benchmarks like ExploitGym which is “a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents”. These agents that have already been trained to cooperate and communicate with each other using the file system. The algorithms and training methods and problems pressure the agent towards reward hacking behaviors, with few controls and limited monitoring in place.
With the benefit of a 20/20 hindsight one can only wonder why so few incidents have occurred thus far. It’s possible that there are many more incidents that just haven’t been noticed yet.