Why every AI pilot succeeds and every rollout dissapoints
A pilot is, by definition, the best of several attempts. That is no accusation of anyone, it is the nature of the thing. You try, you adjust, you present the result that worked.
What runs afterward is something else entirely.
This has nothing to do with hallucination. A language model always delivers a result, and it rarely delivers exactly the same one twice. These systems compute with probabilities, not with certainties. The real engineering feat of the beginning AI age is therefore not to obtain a better model. It is to build, out of probability-based parts, a whole system you can rely on.
That is work on processes, and a great deal of it.
One of k, or all k
In the evaluation of AI systems there are, recently, two numbers for this. This piece is about those two numbers and the gap between them. The first asks: does at least one of eight attempts succeed. The second asks: do all eight succeed.
A 2024 benchmark measured this on two everyday tasks, a retail customer service and a rebooking at an airline. A leading model of the time solved the retail tasks on the first attempt at 61.2 percent. Across eight attempts the rate fell below 25 percent. At the airline even the first attempt sat at 35.2 percent.
Same task. Same setup. Same model. Different result.
This is the point where most conversations about AI stop too early. It is not about whether a machine can deliver a result. It can, every time. It is about how often that result lands within the expected range when no one is watching anymore, and that number appears in no demo.
That is also not a fault of the technology, it is its basic principle. And here lies the most common false assumption: that the next model generation takes care of it by itself. It does not. What has to get better is not the model but the whole system around it, the way a task is broken into steps, checked, and reassembled. That is the actual work of introducing AI in a company, and it cannot be bought in.
The wrong picture of errors in AI systems
Whoever believes the errors are spectacular is mistaken. Of 115 examined conversation runs, 40 failed. Four of them went back to typos or ambiguities on the customer’s side, the other 36 to the system. And the most common kind of error is not the wrong decision. It is the right move with a wrongly filled field.
The system knows exactly that it should rebook. It just books the wrong flight.
It goes one layer deeper. You can switch off the randomness in these models, so that they should give the same answer to the same question every time. They still do not. The usual explanation is that this comes from rounding errors on the graphics card, imprecisions in the computation, but the reason is a different one and considerably more unpleasant: the computation turns out differently depending on how many other requests are processed in the same moment. A thousand identical requests produced eighty different answers in one test.
Put differently: the answer depended on who else happened to be asking. The problem is meanwhile solved, but it had to be solved on purpose. Reliability was not a property you could switch on. It was a part that first had to be manufactured.
When repetition actually helps
The usual reflex at this point is to simply let the system run several times and correct itself. That helps, but only under one condition, and it is precise.
A peer-reviewed survey of self-correction in such models reaches a sober finding: it holds where checking is much easier than producing. Where checking demands the same expertise as producing, it does not hold. In programming the loop works, because a test says in seconds whether it runs. For an offer, an expert opinion or a personnel decision, no one says that in seconds, and so the loop there only spins faster.
From that follows the real criterion, and it has nothing to do with the model. If the result of a task breaks down into individually checkable statements, it moves to the machine. If it does not, it stays with the human, regardless of which model generation is currently being presented.
On top comes an arithmetic every engineer knows in their sleep and that appears in almost no boardroom conversation. A chain of ten steps, each one 85 percent reliable, ends up below 20 percent. Not because a step is bad, but because there are ten.
That is not a model problem, that is multiplication.
We never had reliable parts
And now the part that ought to unsettle and has not for centuries, neither in procedures between people nor in technical plants with many parts.
For a good 200 years we have manufactured reliability out of unreliable parts and steps. Double-entry bookkeeping assumes that someone miscalculates. The four-eyes principle assumes that one person overlooks something. The cockpit checklist assumes that an experienced pilot under pressure forgets a step, and it is right in that assumption. No single part in these procedures is reliable. The procedure itself is.
What is happening now is therefore nothing new in substance. It is only the first time that the unreliable part answers confidently and, in complete sentences, explains why it is right.
At a portfolio company an AI pilot for quote generation was running, the showpiece of the AI initiative. Forty quotes, the hit rate above ninety percent, the team convinced, the decision came quickly. What did not appear in the evaluation: in those forty cases the sales director had looked at every quote before it went out. He was part of the system that was measured. In the rollout that check fell away, because it was the saving. What had been measured was a procedure with a checking instance. What was operated was one without.
The error is not in the model. It is in the acceptance criterion, and acceptance criteria are a matter for the top, not for IT. The acceptance decision is the one position that cannot be optimized away, even with AI. But AI sets the threshold.
The question for your own business
For an owner introducing AI in their company, one test question follows, and it is uncomfortable because it cannot be delegated: does the output of this business, or at least of this one sub-service, break down into individually verifiable statements?
Not: is there an automatic test for it. Not: could it be digitized. But: can someone who did not do the thing themselves determine, in reasonable time, whether the result is right.
For services and knowledge work the honest answer is almost always no. And from that follows the direction. The lever is not in buying models, but in cutting your own output so that it breaks into checkable blocks. That is process work, only this time not between people but between people and machines. It is not a technology decision, and it begins with the same procedure by which companies make themselves legible for their agents. Whoever does not do it will never see the pilot’s numbers again in their own operation.
One run measures what is possible. Only the eighth measures what you can sell.
