Where does the 36% figure come from?

It is arithmetic, not a study: 0.95 raised to the power of 20 is 0.358. A twenty-step workflow in which every step is right 95% of the time completes correctly about 36% of the time. At 99% per step it is 82%, and at 99.5% it is 90%. The point of stating it is that per-step reliability is what gets demonstrated in a pilot, and end-to-end reliability is what the business actually experiences. Nobody computes the second number out loud.

Why is prior authorization the most important example?

Because it is a natural experiment at national volume, reported from actual annual transaction counts rather than a survey. The CAQH Index shows that a decade of mandates, financial incentives and standardisation moved mechanical transactions to 96-99% electronic while prior authorization, the judgement-laden one, stayed at 31-37%. Same industry, same incentives, same decade. The variable that predicted the outcome was whether the step required a judgement. One trap worth knowing: "fully electronic" measures the channel, not whether a human adjudicator intervened.

So should we not automate anything with exceptions?

No — almost everything has exceptions. The argument is that the exception rate should be counted from logs before the build, not discovered after it, and that the design question is which cases may be decided without a person. EY, selling RPA implementations in 2016, recommended automating the lowest-value 70% and leaving 30% to humans as an initial target. That is a reasonable starting posture, and it was written by the vendor.

How should we decide what the system is allowed to do on its own?

Route on blast radius and reversibility rather than on model confidence. Verbalized confidence is a weak signal, and the business can answer "how bad is this if it is wrong, and can we undo it" without the model. Drafting an internal memo and changing a customer contract may run on the same underlying model while justifying entirely different levels of machine authority.

Is there evidence that designing the escalation path improves outcomes?

Not at deployment scale, and we think saying so is worth more than pretending otherwise. No study links up-front exception design to deployment outcomes. What exists is lab-scale learning-to-defer research showing a well-designed handoff can beat either party alone — one demonstration reached 91.3% for an evidence-assisted hybrid against 87.7% for the model alone. That literature also finds that supporting a human working a deferred low-confidence queue is largely unstudied, and that some forms of AI assistance stop helping and start harming as reviewers improve with practice.

August 24, 2026AIssential

Your Pilot Ran on the Standard Case

Your business runs on the exceptions — and somebody already measured the difference

TL;DR — Key Takeaways

A twenty-step workflow at 95% per-step reliability is correct end to end 36% of the time. Enterprise operations need failure rates under 1-5%. That distance is the pilot-to-production gap.
Agents that succeed once often cannot repeat it: tau-bench measured pass^8 below 25% in retail, and CRMArena-Pro found ~58% single-turn dropping to ~35% multi-turn.
The cleanest evidence is national-scale: after a decade of mandates, US health insurance moved eligibility verification to 96-99% electronic and claim submission to 98%, while prior authorization stayed at 31-37%. Incentives did not move the judgement layer.
Real work is mostly exception: 18.4% of invoices, only 12% of life insurance applications eligible for automated decision (and 83% of those still touch a human), and real process logs showing up to 80% trace uniqueness.
Air Canada lost in tribunal over a chatbot that got the policy shape right and the exception timing wrong. Damages: $650.88. The size is the point.
Honest limit: no study links up-front exception design to deployment outcomes. We say so rather than claiming the evidence we would like to have.

Your pilot worked. It handled the cases you gave it, the demo landed, and everyone in the room agreed it was impressive. Six months later it is still a pilot.

The usual explanations are change resistance, data quality, or the model. Occasionally one of those is the answer. More often the pilot ran on the standard case and the business runs on the exceptions, and nobody measured the difference before committing.

This is not a soft claim about organisational readiness. It is arithmetic, and it has been measured at national scale.

The arithmetic nobody does out loud

Take a workflow of twenty steps. Suppose each step is right 95% of the time — a number most teams would be pleased with.

End to end, that workflow is correct 36% of the time.

At 99% per step it is 82%. At 99.5% it is 90%. Enterprise operations typically need failure rates under 1–5%, which means per-step reliability in the high nines. The distance between "impressive in a demo" and "safe in production" is not a gap in enthusiasm. It is several orders of magnitude of per-step reliability, and it compounds silently because nobody sees the twenty-step number.

The measured evidence says this is exactly where systems break:

Agents that succeed once often cannot repeat it. τ-bench (Sierra AI and Princeton, arXiv:2406.12045) scored agents against the final state of a database rather than against a rubric, and introduced pass^k — the probability of succeeding k times in a row. Verbatim: "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."
Multi-turn is where it goes. CRMArena-Pro (Salesforce AI Research, arXiv:2505.18878), built on 4,280 expert-validated task instances in a real Salesforce org, measured roughly 58% success on single-turn tasks and about 35% multi-turn. Note who published it: research unflattering to the category its own employer sells.
The headline horizon is a coin flip. METR (arXiv:2503.14499) measured how long a task a model can complete, and found the 80%-reliability horizon is about five times shorter than the 50% one. Every "AI can now do multi-hour tasks" headline quotes the coin-flip threshold, not one any operations team would sign off on.

Read that against GDPval (OpenAI, arXiv:2510.04374), where one-shot deliverables graded blind by professionals with ~14 years' experience reached near-expert quality. Both results are real, and together they are the whole pattern in two sentences: one artifact, one shot, graded on quality — that is the demo. Repeated multi-step execution against state — that is production. Models are strong on the first and weak on the second.

How much of real work is the exception

Here is the part that decides whether a workflow is worth automating, and almost nobody measures it before starting.

Health insurance, at national transaction volume. The CAQH Index reports actual annual transaction counts across plans, providers and clearinghouses — billions of them, not a survey. After a decade of mandates, financial incentives and standardisation, the mechanical transactions went nearly all the way: eligibility verification 96–99% fully electronic, claim submission 98%. The judgement-laden one did not move. Prior authorization sits at 31–37%, one of the lowest in the index, gaining about three points a year.

Same industry, same mandates, same money, same decade. Mechanical work: 98%. Judgement work: a third. That is the cleanest natural experiment available on where automation stops, and the variable is not technology or willingness. It is whether the step requires a judgement.

Accounts payable. Ardent Partners' State of ePayables 2025 (n=204) found 18.4% of invoices are exceptions and only 35.4% go straight through. Best-in-class organisations run 11.1% exceptions against 20.9% for everyone else — better, not solved. The report's own line is the one to keep: "the same exception problem will have to be solved again (and again)." The respondents to a survey about payables automation skew more automated than the field, so treat 18.4% as a floor.

Life underwriting. Gen Re's 2024 US survey of accelerated underwriting found that only 12% of applications are eligible for a fully automated decision — and that 83% of even the automation-eligible ones still get a human underwriter. Inside the pre-qualified easy population, five in six cases still touch a person.

And the shape of real process data. An analysis of 98 public event logs (TU Munich, BPM 2025) found real processes containing up to 22,632 distinct variants. In the widely used Sepsis Cases log, 1,050 traces produce 845 variants — roughly 80% trace uniqueness. Four in five real cases follow a path no other case follows. A pilot built from the modal path has not seen the process.

What this costs when it goes wrong

Moffatt v. Air Canada, 2024 BCCRT 149, is the cleanest documented example on record and it is small on purpose. Air Canada's chatbot told a passenger that bereavement fares could be claimed retroactively within 90 days. The actual policy says they cannot. Air Canada argued in tribunal that the chatbot was "a separate legal entity that is responsible for its own actions." The tribunal's reply: "This is a remarkable submission… It makes no difference whether the information comes from a static page or a chatbot." Negligent misrepresentation. Damages: $650.88.

The system got the general shape of the policy right and the exception's timing rule wrong. That is the failure mode. Not a hallucinated fact anyone would catch — a plausible answer about a low-frequency carve-out, delivered with the same confidence as the correct ones.

What to do about it — and what nobody can prove yet

The design principle that follows is not "add a human in the loop" as a slogan. It is a question about which cases get to be decided without one, answered before the build rather than after.

Three questions worth answering on a real workflow:

What fraction of volume is the exception? Not estimated — counted, from logs. If nobody can produce the number, that is the first finding.
What happens to the exception when the standard case is automated? Removing the routine 70% does not leave the remaining 30% unchanged. It removes the routine work that kept people fluent in the domain, and the exception queue is now all that is left.
How much human work remains after the system has run? If a senior person has to reconstruct the whole case before approving what the system produced, very little work has actually disappeared. This is the measurement that separates a productivity gain from a productivity theatre, and it is almost never taken.

Now the honest part. The obvious conclusion from all of this would be that designing the escalation path up front is what separates deployments that work from deployments that stall. We would like to claim that. We cannot: no study links up-front exception design to deployment outcomes. What exists is lab-scale learning-to-defer research showing that a well-designed handoff can beat either the human or the model alone — in one demonstration, an evidence-assisted hybrid reached 91.3% against 87.7% for the model alone. That literature also carries two findings that cut against easy optimism: how to support a human working a queue of low-confidence deferred cases is largely unstudied, and as reviewers get better with practice some forms of AI assistance stop helping and start harming.

One principle from that work does survive contact with practice, and it is worth carrying: route on blast radius and reversibility, not on model confidence. Verbalized confidence is a weak signal. "How bad is this if it is wrong, and can we undo it" is a question the business can answer without the model's help.

The consultancy that said this in 2016

There is a useful precedent, from the automation wave before this one. EY's 2016 RPA guidance — written by a firm selling RPA implementations — recommended: "Automating 70% of a process that is the lowest-value, and leaving the high-value 30% to humans is a good initial target."

The leading vendor of the previous automation wave recommended leaving roughly a third to people, as a design target, on day one. The wave that followed produced Deloitte's 2018 finding that only 3% of organisations had scaled past 50 robots, with 33% still piloting, and "process standardisation is the top challenge across all stages."

The technology is far better now. The distribution of real work is the same as it was.

The one-line version

Your pilot proved the model can do the work. It did not prove your process is doable. Those are different claims, and only the second one has a budget attached.

If you have a workflow where AI should be paying off and is not, the first thing worth measuring is not the model. It is what fraction of your volume never looked like the demo.

Make the AI decision you can defend.

Try AIssential for free →