Measuring agents honestly: pass^k, error bars and time horizons

“Our agent solves 81% of tasks.” It is the sentence at the top of every agent launch post and most internal reviews, and it answers a question almost nobody is asking.

A customer-support team does not care whether the agent can resolve a refund. They care whether it resolves every refund, day after day, without the 1-in-6 conversation that ends with a wrong payout. An engineering lead comparing two agents does not care that one scored two points higher; they care whether those two points are real. And someone deciding whether to hand an agent a four-hour task does not care about its score on five-minute ones.

Three ideas from recent evaluation research answer those questions. None of them is complicated. Together they change what a good agent report looks like.

1. Reliability: pass@k is not pass^k

Language-model agents are stochastic. Run the same agent on the same task twice and it may take different paths, and sometimes reach different outcomes. A single-attempt success rate averages that away.

Two metrics put the variation back, and they point in opposite directions:

• pass@k — the probability that at least one of k attempts succeeds. It comes from code generation, where you can run several candidates and keep whichever passes the tests. It goes up with k.
• pass^k (read “pass hat k”) — the probability that all k attempts succeed. It was introduced with τ-bench, Sierra’s 2024 benchmark of agents serving simulated customers through tools and policies. It goes down with k.

Which one matters depends on whether you get to retry. A coding agent whose output is checked by tests lives in a pass@k world. A support agent talking to customers lives in a pass^k world: every conversation counts, and there is no “keep the best of eight”.

Both have simple unbiased estimators when you run each task n times and observe c successes:

from math import comb

# 8 runs of the same agent on each of 6 support tasks; successes per task
n = 8
successes = [8, 8, 7, 7, 6, 3]

def pass_at_k(c, k):   # at least one of k attempts succeeds
    return 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(c, k):  # all k attempts succeed
    return comb(c, k) / comb(n, k)

print(" k   pass@k   pass^k")
for k in (1, 2, 4, 8):
    at = sum(pass_at_k(c, k) for c in successes) / len(successes)
    hat = sum(pass_hat_k(c, k) for c in successes) / len(successes)
    print(f"{k:>2}   {at:6.0%}   {hat:6.0%}")

Output:

 k   pass@k   pass^k
 1      81%      81%
 2      93%      69%
 4      99%      54%
 8     100%      33%

The same agent, the same runs, three very different stories. Allowed eight tries, it “solves everything”. Required to succeed eight times in a row, it is reliable on only two tasks in six. The original τ-bench results showed exactly this pattern for the strongest models of the time: respectable single-attempt scores, with pass^8 in the retail domain falling below 25%.

What to do with it: measure pass^k at a k that matches your exposure — if a workflow runs the same kind of task a hundred times a day, pass^1 tells you almost nothing about how many of those hundred go wrong. Then look at the tasks with low c/n. Those are where reliability work pays, and they are usually a handful of specific failure modes rather than general weakness.

2. Honesty: put error bars on every number

An eval score computed on a finite set of questions is an estimate, and estimates have uncertainty. Leaderboards almost never show it.

Anthropic’s 2024 paper Adding Error Bars to Evals sets out a short list of recommendations that are now widely cited:

1. Report the standard error of the mean with every score. For a pass rate p on n independent questions it is about sqrt(p(1-p)/n), and a 95% interval is roughly ±1.96 standard errors.
2. Use clustered standard errors when questions are related. Many evals contain groups of questions about the same passage, the same repository or the same environment. Those are not independent, and treating them as if they were makes the error bars too narrow — sometimes much too narrow.
3. Compare models with paired differences. When two systems answer the same questions, analyse the per-question difference. Much of the noise is shared — hard questions are hard for both — so a paired comparison detects real differences that two separate intervals would hide.
4. Reduce variance within questions by sampling several answers per question, or by scoring with probabilities rather than single samples.

The practical effect is to make some leaderboard gaps evaporate. At around 80% accuracy on 500 questions, the standard error is about 1.8 points, so the 95% interval on each score is roughly ±3.5 points. Two agents separated by two points on that benchmark have not been shown to differ at all — unless a paired analysis says otherwise.

3. Capability: how long a task can it finish?

The third idea addresses a different question: not how often the agent succeeds, but on how big a job.

METR, an independent AI-evaluation organisation, measures this with a time horizon. They assemble a large set of software and research tasks, time how long skilled humans take to complete each one, and then fit, for each model, the relationship between task length and the model’s probability of success. The 50% time horizon is the human task length at which the model succeeds half the time.

METR’s 2025 paper found that this horizon had grown exponentially for six years, doubling roughly every seven months. Their more recent measurements suggest the doubling has been faster since 2024, and the newest frontier models now sit beyond the range their task suite can measure reliably — METR flags estimates above about 16 hours as unreliable with its current tasks.

Three things make the time horizon useful beyond the headline:

It is a common unit. “Completes tasks that take an expert two hours” is comparable across benchmarks and model generations in a way that “scored 64 on benchmark X” is not.

The 80% horizon is the one to plan with. Success falls off with task length along an S-shaped curve, so the length at which an agent succeeds 80% of the time is several times shorter than the 50% length. If you need work done reliably, the 50% number overstates what you can delegate.

It measures the right failure. Long tasks fail less from lack of knowledge than from compounding errors — a wrong assumption early, a lost thread of state, a loop that never ends. The horizon captures that in one number, and it is the reason long-horizon failure modes deserve their own engineering.

Putting it together: what to ask for

The next time an agent evaluation lands on your desk — from a vendor, a paper, or your own team — ask for four things:

1. pass^k at a realistic k, alongside the single-attempt score, with the number of runs per task stated.
2. An interval, not a point, with clustering accounted for and, for comparisons, a paired difference.
3. Success by task length, or at least the length distribution of the tasks evaluated, compared with the tasks you need done.
4. The failures. A list of the tasks with the lowest per-task success rate, because that is where both the risk and the fix live.

An agent report with those four things is harder to make look good. That is precisely what makes it worth reading.

What to take away

• A single-attempt success rate hides consistency, uncertainty and task length.
• pass@k rewards retries; pass^k measures reliability. Customer-facing agents live in the pass^k world, and it falls fast with k.
• Report standard errors, cluster related questions, and compare systems with paired differences. Small leaderboard gaps are often noise.
• METR’s time horizon measures how long a task an agent can finish. It has been doubling every several months — and the 80% horizon is the one to plan around.

For how improving against an eval can itself mislead you, see hill-climbing your evals.