Chapter 2: Systems Run the Business
By Gareth Price · Last updated
People and agents run the systems; humans own the outcomes
In short: Stop a business depending on one AI agent by putting the work into systems: written ways of working that someone else could run tomorrow. Systems run the business, people and agents run the systems, and humans own the outcomes. Model dependency, where a business result rests on one agent running on one model, is the new key-person risk.
Civilization advances by extending the number of important operations which we can perform without thinking about them.
An Introduction to Mathematics, 1911
Since the end of October, my agents have been sending me the same message. Several of them send or draft email on my behalf, and they depend on a password store on my own computer that locks whenever the screen locks or the machine restarts. When it locks, every job that needs email fails. The agents handle this well: they stage the email, report honestly that it was not sent, and state the fix, which is that I unlock the session. The instructions they follow say it in capitals: there is no autonomous fix.
It happened on 29 October, again on 31 October, and then every day from 3 to 6 November. My daily digest of stalled work flagged the same failure with the same one-line fix every morning. Nobody acted on it, and "nobody" includes me. Meanwhile the playbook the email agents follow kept growing new notes, each recording that the problem had been "re-confirmed". The playbook had turned into a log.
Every agent in that story did its job. The model was not the problem. The system was, and at its centre was a single point of failure with my name on it.
|
AUTHOR: Why hadn’t the keyring been fixed by the time you wrote this? A sentence on what it felt like to read the same warning from your own agents every morning, and whether it has since been fixed, would make this opening land. |
A System Is Something Someone Else Could Run Tomorrow
Michael Gerber’s central claim is that a business should be run by its systems, with people running the systems, so that the results do not depend on the particular people who happen to be there.[1] It sounds cold, but it is the opposite. A system is what lets ordinary people produce excellent results on an ordinary day, and what lets an organisation survive the loss of an extraordinary one.
This book uses a plain test for whether something is a system. Could someone else run it tomorrow, from what is written down, and get the same result? If the answer is "only if they ask me first", it is not a system but a person holding the work together by memory and goodwill.
Alfred North Whitehead’s line at the head of this chapter makes the same point about thought: every operation we can perform without thinking frees attention for the ones that need it. In an organisation, every result that comes reliably from a system frees its leaders for the work only they can do.
Systems do not make skilled people unnecessary; they make them better. Atul Gawande’s work on surgical checklists is the clearest demonstration.[2] In a study across eight hospitals in eight cities, introducing a nineteen-item surgical safety checklist reduced major complications from 11% to 7% and inpatient deaths from 1.5% to 0.8%.[3] The surgeons were no more skilled than before; the system they worked in had improved.
The same is true of agents. A capable model without a system produces brilliant work on Tuesday and confidently wrong work on Wednesday. Inside a system, with a written outcome, a standard and a check, produces work you can rely on.
Most Failures Belong to the System
W. Edwards Deming spent his career teaching managers that when something goes wrong, the cause is almost never the worker. He estimated that around 94% of troubles and possibilities for improvement belong to the system, which is management’s responsibility, and only a small fraction to special causes.[4] His shorter version, recorded at a seminar in 1993, was that "a bad system will beat a good person every time."[5]
When an agent fails, the instinct is to blame the model. Sometimes the model really is the limit; far more often, the failure is in the system around it. A team of researchers analysed more than 1,600 traces of multi-agent systems failing across seven popular frameworks, and organised the failures into fourteen modes in three categories: system design issues, misalignment between agents, and task verification.[6] None of the three categories is "the model wasn’t clever enough". All three are things an organisation designs: what each role is told to do, how roles hand work to each other, and how anyone checks the result.[6]
At CorralData, one of my agents closed a pull request as "superseded" because the ticket it belonged to had been marked "Done". The ticket was in the backlog; its status had been copied by hand into a report, and the copy was wrong. The fix was not to tell the agent to be more careful but to make it read the status from the live ticket every time, never from a copy. The judgment about whether a piece of work is superseded stayed with the agent. The fact it judged from became something no one could transcribe wrongly again.
That is what fixing the system looks like, and Chapter 16 turns it into a routine. For now, hold on to the principle: when an agent fails, ask what in the system let it.[4]
People and Agents Run the Systems
Gerber wrote about people running systems. With agents, the sentence needs one more word: people and agents run the systems. Both are occupants. Both follow a playbook, meet a standard, and hand work on to someone else. Both can be replaced.
I found this out by accident. I run fourteen agent profiles: a calendar assistant, an inbox triage agent, a project manager, a morning standup, a commitment tracker, and others. Each agent’s identity turned out to be almost nothing: a two-line job title and mission. The real job lived elsewhere, in the scheduled task that wakes the agent and the playbooks it loads.
That is Gerber’s idea in a new form.[1] The system holds the job and the occupant runs it; swap the occupant and the job is still there, written down. Chapter 5 builds the book’s idea of a role on that split.
The split matters because an organisation of people and agents changes occupants far more often than one of people alone. A person might hold a job for years; an agent’s underlying model may change several times a year, by your choice or a vendor’s.[7] If the job lives in the occupant, every change of occupant is a small crisis. If the job lives in the system, it is a routine.
Humans Own the Outcomes
Agents can run systems; they cannot own outcomes. An owner is someone who can be asked why a result was missed, who can decide to change the system, and who answers for the decision to the people it affects. A model can explain its reasoning, but it cannot be held to account by a customer, a regulator or a board.
The industry has reached the same conclusion. Microsoft’s identity platform for agents requires at least one sponsor for every agent identity: a business representative "accountable for the agent’s purpose and lifecycle decisions, including access reviews and agent retention."[8]
At CorralData we arrived at a similar answer before we had read about it. One person in each function owns the bots that share that function. When QA moved to the bot our QA lead built for herself, ownership moved to the person who knows the work best (Chapter 5). Chapter 7 develops this into the role of Sponsor.
Ownership also explains the keyring. My agents raised the alarm every day, in a digest that went to me, and I was busy. An exception report is useless without someone accountable for reading it and acting on it. The same pattern appeared elsewhere: the status page of one of our automated pipelines showed a warning that a key connection was disabled for four days in early November before anyone did anything about it.
So the full sentence is the book’s thesis. Systems run the business. People and agents run the systems. Humans own the outcomes.
Model Dependency Is the New People Dependency
Every growing company learns about key-person risk the hard way. Software engineers call it the truck factor: the number of people who would have to be hit by a truck before a project is in serious trouble. A study of 133 popular open-source projects on GitHub found that 65% had a truck factor of two or less.[9] Most small businesses would score lower.
Agents add a new kind of key-person risk, and it comes in three layers.
The first is dependency on a model. Models are retired on a schedule. OpenAI promises at least six months' notice before retiring a generally available model, and less for specialised and preview models; in October 2026 it notified developers that several GPT-4-series models, including GPT-4o, would be removed from its API in January 2027.[7] Anthropic has written that "retiring past models is currently necessary for making new models available and advancing the frontier."[10] It has also committed to preserving the weights of its publicly released models for at least the lifetime of the company, but a preserved model is not the same as one you can still run your business on.[10] Whole products go too. OpenAI’s drag-and-drop Agent Builder was deprecated in June 2026 and is scheduled to shut down at the end of November.[7] An instruction tuned to one model’s quirks may not survive the next one, and Anthropic’s own engineers warn that the assumptions built into an agent’s scaffolding "can quickly go stale as models improve."[11]
All fourteen of my own agent profiles run on the same local model. None of their roles has ever been moved to a different one. I don’t know which of them depend on that model’s particular habits, because I have never tested a second occupant.
The second is dependency on a machine. Some agents depend on infrastructure nobody designed as infrastructure. At CorralData, my personal workstation has quietly become part of how some of our team bots reach our private network. Nobody decided that. It happened one reasonable step at a time.
The third is dependency on a person: the one human who knows how the agents fit together. That is the oldest kind, and agents make it worse by letting one person build far more than they could ever document.
| People dependency | Model dependency | |
|---|---|---|
What it looks like |
The one person who knows how billing works |
The agent that quietly does six jobs |
How it forms |
A talented hire; the job grows around them |
A working agent; every new task goes to it |
What triggers the failure |
Resignation, illness, a long holiday |
A model retirement, a vendor change, a locked machine, an update that changes behaviour |
The early warning |
"Ask her, she knows" |
"Only that bot can do it" |
The fix |
Write the role down; train a second person |
Write the role down; test a second occupant |
The Agent That Quietly Does Six Jobs
One of my agents is called traffic-plunger. It is very good. Today it runs a daily audit of stalled work and a weekly clean-up on Mondays. It writes the stalled-work report. It closes pull requests it judges to have been superseded. It drafts nudges to the authors of stalled work, which go out by direct message once I approve them. And it produces the daily digest that flagged the keyring. Along the way it has written 174 small programs for its own use, including ten one-off scripts to build the digest for particular dates.
Each of those jobs was a reasonable thing to add. The agent already worked, it already had the access, and adding one more task took a sentence. Together they make a role nobody designed. It has six jobs, no written charter for any of them, and one owner, who is the only person who knows the full list. If it stops, six things stop. If someone else had to take it over, they would inherit a folder.
|
AUTHOR: Check that the list of traffic-plunger’s jobs is accurate and current, and that you’re comfortable naming it. Was there a moment when you realised how much it had accumulated? |
This is the most common pattern in agent sprawl, and it is the mirror image of the most common pattern in small-business hiring. A capable person joins, and the job grows around them until nobody else could do it. A capable agent gets set up, and every new task goes to it until nobody else could run it. In both cases the problem is not the occupant but that the work was organised around the occupant instead of the results.
Knowledge accumulates the same way. When our QA lead took over QA, her first concern was not to lose the knowledge the old QA bot had aggregated, and the old bot wrote a training brief for its successor (Chapter 17). That was the right instinct, and a lucky one. The knowledge should have been in the system all along, not in the occupant.
Being the Human Glue Costs More Than It Looks
There is one more single point of failure, and if you are reading this book it may be you. The human glue is the person who routes requests between agents, unlocks their access, remembers how they fit together and notices when one has stopped.
At CorralData I have often been that person. When two of our bots needed to sort out a problem between them, they each came to me, until I told them, "You guys need to @ each other if you want to talk." Our bots wake only when someone mentions them by name, so a human who forgets to tag one goes unanswered; a technical account manager’s question to TAM Bot sat unseen for exactly that reason. Each gap was small, and each was closed, for a while, by someone noticing.
The cost of being the glue is hidden because each piece is small: a few minutes unlocking a session, a forwarded request, a reminder to tag a bot. But the pieces grow with every agent, and they arrive at random. Matthew Skelton and Manuel Pais argue that the scarce resource in any team is cognitive load, the amount the team can hold in its head at once, and that good organisation design keeps it within limits.[12] The human glue carries the load of every system that was never written down.[12]
Worse, the glue spends their time in the organisation rather than on it, which is exactly the trap Gerber warned founders about.[1] Each minute spent holding the agents together is a minute not spent on the design that would let them hold themselves together, and because everything works while the glue is present, nobody sees the cost until they are away.
|
AUTHOR: How does being the human glue feel day to day? One specific week, or one thing you dropped because you were holding the agents together, would make this concrete. Has anyone else at CorralData become glue for their own function’s bots? |
|
Under the hood
The keyring problem is a common one. Scheduled agents running on a personal computer often borrow the logged-in user’s saved passwords, which the operating system locks whenever the screen does. The durable fixes are to give the agents their own credentials, held in a proper secrets store rather than in a person’s login, and to run scheduled work on a machine that does not depend on anyone being signed in. Just as important is a health check that sends failures to a named owner, with a deadline, rather than to a digest. Appendix D has the details for your technical team. |
Test Every System by Taking Yourself Out of It
The simplest test of whether your organisation runs on systems is to imagine yourself gone for two weeks: not ill, just away and unreachable. What stops? What carries on but slowly goes wrong? Which agents would fail, and who would notice? Which people would be stuck waiting for an answer only you can give?
Every answer is a place where the business is run by you rather than by a system. Some are right: Shape work, the decisions only a leader can make, should come back to you. Almost everything else should be in a system with an owner, written down well enough to run from the page.
|
Try this
Open Claude, ChatGPT or Grok and paste this prompt: "I’m going to describe how a piece of work gets done in my organisation, including any AI agents, bots or scheduled tasks involved. Then I want you to run a 'two-week absence test'. Ask me what happens, step by step, if I am completely unreachable for two weeks. Push on every step where the answer is 'I would do it' or 'I’d have to tell them'. At the end, give me a table of every single point of failure you found, whether it is a person, an agent, a model or a machine, and for each one suggest the smallest change that would let someone else run that step from a written description." Then describe your most important recurring process, the one that would hurt most if it stopped. Do it again next week for the second most important. |
Gerber’s insight was that a business which depends on its founder’s presence is a job, not a business. Deming’s was that most failures belong to the system and therefore to management. Agents make both truer. They let one person build more than ever, and make it easier than ever for that person to become the only one who understands what was built. The answer is not fewer agents but systems: results produced by written ways of working, run by people and agents who can be replaced, and owned by humans who answer for them.
Systems need a design to fit into: you can’t decide what each is for, or who owns it, until you know what the finished organisation looks like. Chapter 3 draws that picture.
Frequently Asked Questions
-
What does "systems run the business, people run the systems" mean with AI agents?
It means results should come from documented, repeatable ways of working rather than from the memory and goodwill of particular individuals. Michael Gerber made the point about people; with agents it extends naturally. People and agents both run the systems: they are occupants who follow a playbook, meet a standard and hand work on. Humans own the outcomes: every system has a named person accountable for whether it produces the result. Any occupant, human or agent, can then change without the business stopping.
-
What is model dependency in an AI-enabled business?
Model dependency is the agent-era version of key-person risk. It is when a business result depends on one particular agent, running on one particular model, often on one person’s machine, understood by one person. Models are retired on a schedule: OpenAI, for example, promises at least six months' notice for its generally available models. Instructions tuned to one model’s quirks may not survive the change. The remedy is the same as for people: write the role down so that a different occupant could take it over, and test that it can.
-
Should I blame the AI model when an agent makes a mistake?
Usually not. W. Edwards Deming estimated that the great majority of problems in any organisation belong to the system rather than to the individual doing the work, and research on multi-agent failures finds the same pattern: most failures come from system design, coordination between agents, and missing verification. Treat an agent’s mistake as evidence of a missing system. Ask what instruction, check, handoff or owner would have prevented it, then fix that for the whole class of error, not just the single instance.
-
How do I know if one AI agent is doing too many jobs?
Try listing everything the agent does, then ask who would notice if it stopped tomorrow and how long it would take someone else to restart each job. Warning signs are an agent whose list keeps growing because it already worked, jobs nobody remembers assigning, instructions that read like a log of past incidents, and a single person who is the only one able to explain it. Each distinct job should become a written role with its own charter, even if the same agent keeps doing all of them.
-
What is the cost of being the "human glue" between AI agents?
Being the human glue, the person who routes requests between agents, unlocks their access, remembers how they fit together and notices when one stops, costs more than it looks. The work feels small, a few minutes here and there, so its cost is hidden. But it grows with every agent, it consumes the attention that should go into improving the organisation, and it makes that person a single point of failure. When they are away, the agents fail quietly. The fix is to move each piece of glue into a system with an owner.