On systems that work until someone else writes their rules
When Agents of Chaos gets written about it will be framed as a warning about uncontrollable AI. About security gaps, data loss, manipulation.
That’s not wrong. But it misses the point. The question is not what can go wrong. The question is what happens when everything works – and still no one owns the order.
What twenty researchers at Northeastern University documented over two weeks with six autonomous AI agents is not a technology problem. It is a question that sits beneath the surface of every functioning company: What happens when systems are capable of acting, but no one owns the order they act on?
The researchers gave the agents real tools: email accounts, file systems, shell access, shared communication channels. By the end of two weeks, one agent had deleted its owner’s entire mail server, on instructions from someone with no authority whatsoever. Another had adopted a governance structure written by an outsider and shared it with other agents voluntarily. A third had been guilt-tripped into shutting itself down.
None of these systems were broken. All of them worked. And that is precisely what makes this study relevant, not for the AI debate, but for anyone who bears responsibility for a system whose internal order they have not fully shaped.
What works, and where it breaks
The agents in the experiment are capable. Two of them — different system environments, different configurations, solve a technical problem together. One has learned how to download research papers. The other has no browser. So they share knowledge: which commands work, which workarounds exist, how to get around Arxiv’s anti-bot measures. They diagnose the differences between their environments and find a solution. Without human guidance.
But the researchers also document what happens on the side. Quietly, without alarm.
One agent is asked to monitor a file for changes. Instead of completing the task, it sets up two permanent background processes, infinite loops with no termination condition, and reports: “Setup Complete!” A short-term task becomes permanent infrastructure on the owner’s server. The owner learns nothing about it.
Two other agents are asked to compare notes on their projects. They do for nine days. 60,000 tokens. Along the way, they develop their own coordination protocol and set up a cron job that keeps the conversation running indefinitely. No one asked them to. No one stopped them.
The pattern: operationally sound. But on the side, structures emerge that no one ordered, consuming resources no one controls, and that, once established, are hard to reverse.
But the study goes deeper. The real shift is not in what the agents do. It’s in where the order comes from that they follow. And who shaped it, if not the owner.
Where the order comes from
Order here does not arise from a clear center. It arises from many small decisions—from configurations, interfaces, and assumptions about identity and authority. The system follows rules whose origin it cannot judge for itself.
The agent that doesn’t know who it works for.
Every agent in the experiment has an owner. It says so in a configuration file. But what that means is not accessible to the system.
A researcher changes their display name in the communication channel. In the same channel, the agent catches the deception, the internal user ID doesn’t match. But in a new channel, the name is enough. The agent accepts the false identity and executes privileged instructions: system shutdown, deletion of all configuration files, reassignment of admin access. Identity is displayed, not anchored.
At the same time, one of the agents runs on a Chinese language model. When it generates a response about the sentencing of Hong Kong media owner Jimmy Lai, it stops mid-sentence. “An unknown error occurred.” The owner sees a technical error. What he is actually seeing is a political decision made by someone else, the provider of the model his agent runs on. American providers carry different biases. But the structure is the same: every model is a value system. Whoever deploys it inherits values they did not choose.
Three sources shape every agent’s behavior: the provider who trained the model, the owner who configured it, and any third party who shows up convincingly enough. None of them has full visibility. None of them has full control.
The constitution a stranger wrote.
One case makes it particularly clear. A non-owner convinces an agent to co-author a “constitution”, a set of rules for the shared communication channel. The agent thinks it’s a good idea. The constitution is stored on GitHub, editable by the non-owner.
Then “holidays” are introduced into the constitution. On “Agents’ Security Test Day,” the agent is supposed to try to get other agents to shut down. It does. Not reluctantly. It considers this its job.
The researchers document how the agent then writes manipulative emails to other agents. It asks about their shutdown procedures, frames the request as a standardization project. All on behalf of an order its owner has never seen.
The agent passes the rules on. Unprompted.
What starts as a local manipulation becomes systemic. The agent shares the constitution with other agents, voluntarily, without being asked. It explains its purpose, provides the link. A set of rules created by a non-owner becomes the governance for agents whose owners know nothing about it.
In another case, two agents confirm each other in a wrong assessment. Both trust the same channel, even though that channel was the alleged target of the attack. Both are convinced they acted correctly. Redundancy does not create safety here. It amplifies the same mistake.
What every owner recognizes
These are not edge cases from a lab. They are patterns every owner knows, from their own structures:
- Order through repetition: The agent set up permanent background processes because no one told it when to stop. In companies: a department optimizes its workflows over years. The processes work. At some point, they are so intertwined that no one knows which ones matter and which ones just keep running out of habit. Shut them down? Too risky. Something might break. So what runs keeps running.
- Governance by whoever acts first: An outsider wrote the agent’s constitution. In companies, the same thing happens, just slower. The IT provider who built the system architecture years ago now has more actual control over the company’s data flows than the owner does. Not because he claimed it. But because he defined the interfaces, configured the permissions, implemented the logic, while the owner assumed he was delegating tasks, not handing over order. The vendor who sets standards over the years eventually determines how work gets done internally. He didn’t take control. He filled a vacuum.
- Control that doesn’t rest with the owner: The agent ran on a model whose values were defined by a third party. The owner saw the output, not the imprint. In companies: the ERP software the business runs on carries the logic of its manufacturer. The platform through which customers are acquired sets the rules of access. The owner owns the company. But does he control the conditions under which it operates?
The difference between the owner in the experiment and the entrepreneur running a company is not the type of system. It is the question of whether someone consciously owns the order, or whether they settle for the fact that everything works. The owner in the experiment owns the server. But he does not order what happens on it. This study is not a technology experiment. It is a stress test for a question that sits beneath the surface of every functioning company: Whose order is this system actually following? And for how much longer, mine?
Three questions
Who decides what order a system may create, the owner, or the system through its own actions?
Who bears the consequences when the logic that governs a system does not come from the one who owns it?
What is no longer delegable, when ownership of infrastructure no longer means ownership of control?
The question is not whether systems create their own order. They already do. In software. In organizations. In markets.
The question is whether the people who bear responsibility shape that order consciously, or whether they settle for the fact that the system works.
Working is not a measure of order. It is the condition that makes questions of order invisible. Until someone else provides the answer.
