When you test an agent setup and measure its success rate, the number you get may not describe what the user experiences. Because succeeding on average and succeeding every time are not the same thing.
Work published by IBM researchers puts a number on that difference. A ReAct agent using GPT-4.1 succeeds on tasks 77.4 percent of the time on average; but when you run the same task five times independently, the share of tasks it succeeds on in all five is only 53.0 percent.
The 24.4 points in between are what the work calls the consistency gap. This guide covers where that gap comes from, how it is measured and how it is narrowed. The method described is not tied to a particular tool; the measurement logic applies the same way in any agent setup.
Why the same task ends differently
The cause lies in how the model decides. A language model's decisions emerge from probability distributions, and the shape of that distribution is not the same at every step.
When the distribution is "sharp", probability concentrates on one option and the outcome does not change from run to run. When it is "flat", several options carry comparable probability and minor fluctuations can change the result.
That difference in distribution matters especially in agent setups, because an agent does not make one decision. It makes a choice at each of dozens of steps, and just one of those being flat is enough to send the whole chain down another path.
- Sharp distribution: the same step results in the same decision on every run.
- Flat distribution: the decision becomes vulnerable to small platform-level movements.
- Even temperature zero does not remove this entirely.
- A single flat step in a task can change the outcome of the whole run.
Why temperature zero is not enough
The common assumption is that pulling temperature to zero makes the model deterministic. In practice it does not.
When two options in a flat distribution carry very close probability, even small differences at the infrastructure level can change which one is selected. Non-determinism comes not only from the sampling setting but from how clear-cut the decision itself is.
Where the gap becomes visible
The consistency gap shows up in production rather than in the lab, and there is a reason: when testing we usually run each task once.
The user, meanwhile, does the same thing over and over. In a flow that produces the same report every week or fills the same form every day, a one-off success means nothing; what matters is the hundredth run giving the same result.
- A single-run test passes over flat-distribution steps without ever seeing them.
- Different runs of the same task can select different tools.
- A decision that changes at one step sends every following step down another route.
- Users describe this as "it works sometimes", the hardest class of bug to chase.
What Pass^k is
The metric the work proposes enters here. Pass^k shows the fraction of tasks where an agent succeeds on all k independent runs.
| Metric | What it says | Framing |
|---|---|---|
| Mean@k | Average success rate | Neutral |
| Pass@k | Succeeded on at least one run | Optimistic |
| Pass^k | Succeeded on every run | Pessimistic |
The authors are explicit: Pass^k is not Pass@k. The latter is the optimistic frame saying "at least one has to land"; the former is the pessimistic frame where all attempts must succeed. What a user experiences repeating the same query resembles the second, not the first.
Reading the number
The gap between 77.4 and 53.0 percent does not come from failing one task. In a task that has to hold across five runs, one unstable step is enough to break the whole series.
Simple arithmetic shows it: a twenty-step task clearing each step at 95 percent reliability looks reasonable on a single run, but the chance of holding across five runs drops fast. As step count rises, consistency gets exponentially harder, which also explains why improving a single step is not enough.
That is why consistency on long agent tasks becomes a matter of task design more than model quality. A flow doing the same job in ten steps can be more reliable than one doing it in twenty.
Which metric suits which job
The choice of metric depends on the nature of the product. In a flow with human review, Pass@k can be sensible; the user can regenerate an output they dislike.
On an automated pipeline, Pass^k is the metric that counts. An agent running overnight succeeding on four of five attempts means one job in five arrives broken in the morning, and that rate is unacceptable for most workflows.
The first task for a product team is therefore to be clear about which metric it reports. Reporting average success and then claiming reliability answers two different questions with one number.
How the gap is detected
The system the work proposes has two stages: diagnosis first, then correction.
On the diagnosis side a Consistency Analyzer runs. It replays decision steps through controlled resampling and finds which steps are unstable.
- A single trajectory is enough; there is no need to rerun the task from scratch.
- The correct answer does not have to be known, so labelled data is not required.
- Only targeted model calls are made rather than full repetitions.
- The cost therefore stays far below running the task five times.
Where unstable steps usually sit
When the diagnosis runs, unstable steps turn out not to be randomly distributed. Certain kinds of work produce far more flat distributions than others.
The most common place is tool selection. If you hold two tools doing similar jobs, the model cannot develop a clear preference about which to call, and the choice varies from run to run.
The second common place is the moment of deciding the work is done. When the completion condition is not clearly defined, the model sets its own threshold, and that threshold does not land in the same place every time.
How the correction works
The second stage transforms the flagged steps into reusable guidelines. Those guidelines are injected into the prompt at inference time.
The logic of the method follows from that: where the model is undecided, you give it a preference. The guideline sharpens a flat distribution artificially; it does not teach the model something it did not know but points a direction between two things it did.
The result is measured: same-task Pass^5 rises by 16 percentage points while Mean@5 accuracy is maintained. Consistency improves without average success dropping.
In total the consistency gap falls from 24.4 points to 12.0. The gap does not close but halves, and on an automated pipeline that makes a visible difference. In a flow running a hundred times a day, that improvement directly reduces the number of jobs needing manual repair.
How to measure it in your own setup
You do not need a special tool for this measurement; what you need is discipline.
- Fix a task set; even fifteen tasks is enough.
- Run each task five times independently, carrying no state between them.
- Record not the average success but the share succeeding on all five runs.
- Write down the difference between the two numbers; that is your consistency gap.
The gap from the first measurement surprises most teams.
The average rate looks presentable in a report, while the consistency rate shows how reliable the product actually is.
What can spoil the measurement
A common mistake when measuring consistency is carrying state between runs. Run five times in the same session and the model sees the previous attempt in its context, making the result artificially consistent.
The second trap is the outside world changing. If an agent pulls data from an API and that data changed between two runs, the source of inconsistency is the environment, not the model.
So the measurement has to happen in a fixed environment: same inputs, same tool responses, no context carried between runs. Otherwise what you measure is not the agent's consistency but the variability of the world. There is another benefit to this discipline: it makes the effect of changes visible. After a prompt edit, average success can stay flat while consistency drops, and a team watching one number will not notice.
Three common mistakes
Teams setting this measurement up for the first time repeat a few errors, and all of them make the result look better than it is.
- Running the five attempts in the same session: the model sees the previous attempt and consistency rises artificially.
- Picking a set of easy tasks: the gap looks closed, but what was measured is the ease of the tasks.
- Keeping the success criterion loose: outputs counted as "roughly right" lump together results that genuinely differ.
- Measuring only once: consistency changes with a model update, and a single measurement goes stale.
The fourth item is the most neglected. When the provider updates the model the distributions shift, and a step that used to be sharp can become unstable; without repeating the measurement you cannot see it.
Where to stop
Zeroing the consistency gap is not the goal; in a probabilistic system it is not possible. The goal is bringing the gap down to what the product can absorb.
That level varies by job. In a text summarisation flow, four of five holding is not a problem; in a flow that initiates a payment refund the same rate is unacceptable. What sets the threshold is not model quality but the cost of an error. Putting that threshold in writing is the most important decision to make before an agent goes into production.