Research note 001
The 95% trap
A 95% success rate sounds like a solved problem. Chain twenty of those together and your agent finishes barely a third of its runs. This is the single most misleading number in agent evaluation.
The arithmetic nobody quotes
Agent capability is almost always reported per step: the model picked the right tool, parsed the response, wrote valid output. Those are the numbers in the eval table, and 95% looks excellent next to a human baseline.
But a real task is not one step. Posting an invoice touches a ledger, a purchase order, a tax code, and a reconciliation check. Closing a support ticket reads a thread, queries an account, applies a policy, writes a response, and updates a CRM. Call it twenty steps — a conservative estimate for anything operational.
If every step is independent and each succeeds with probability p, the run completes with probability pn. That exponent is the whole problem.
| Per-step | 5 steps | 10 | 20 | 50 | 100 |
|---|---|---|---|---|---|
| 95% | 77.4% | 59.9% | 35.8% | 7.7% | 0.6% |
| 99% | 95.1% | 90.4% | 81.8% | 60.5% | 36.6% |
| 99.9% | 99.5% | 99.0% | 98.0% | 95.1% | 90.5% |
| 99.99% | 100% | 99.9% | 99.8% | 99.5% | 99.0% |
Read the first row again. 95% per step is a 36% task. Two runs in three need a human. The model isn't broken and the benchmark isn't wrong — the metric was simply never measuring what you cared about.
What reliability actually has to be
Invert the question. Suppose you want a run to complete 90% of the time — still not good enough to leave unattended, but a plausible internal bar. What per-step accuracy does that demand?
| Task length | Required per-step accuracy |
|---|---|
| 10 steps | 98.95% |
| 20 steps | 99.47% |
| 50 steps | 99.79% |
| 100 steps | 99.89% |
To get 90% completion on a hundred-step task, every individual step must work 99.89% of the time. That is roughly a fiftyfold reduction in per-step error from where 95% sits — and no announced increase in model capability closes a gap of that shape. It is an engineering and evaluation problem, which is why we think a lab has to attack it directly rather than wait for the next checkpoint.
Independence is a generous assumption
The table above is optimistic. It assumes failures are independent, and in practice they cluster: a misread schema at step 3 poisons steps 4 through 9, and one malformed identifier propagates until something downstream rejects it. Correlated failure makes the real curve worse than the arithmetic, not better.
It also assumes failures are visible. The dangerous case is the step that returns something plausible and wrong — a number in the right format, in the wrong column. Those don't register as errors at all; they register as completed work, and they surface weeks later in a reconciliation.
So measure the thing you want
Our position is narrow and, we think, uncontroversial once stated:
- Count completions, not steps. A run either delivered the outcome in the real system or it didn't. Partial credit is not credit.
- Report task length alongside accuracy. An accuracy figure without a step count is unfalsifiable.
- Instrument recovery separately. How often the agent noticed a failure and re-planned around it is a distinct, more useful number than raw accuracy — and it is the one that compounds in your favour.
- Treat silent-wrong differently from loud-wrong. They have different costs and demand different defences.
making each step slightly better — it's making failed steps survivable.
An agent that recovers turns a multiplicative penalty into an additive one.
That is the entire reason recovery, not retries is our first engineering priority.
None of this argues that models aren't improving quickly. It argues that the number most often used to claim agents are ready is the wrong number, and that the gap between a capable model and a deployable agent is measured in orders of magnitude of reliability rather than points of benchmark score.
We'd rather be honest about the size of that gap and work on it than describe it as almost closed.
NEXT — NOTE 002