Agent benchmarks & eval-driven dev

At 3:07 a.m., your support agent sends a cheerful message: “Your duplicate charge has been refunded.”

The customer checks their bank account. Nothing happened. Worse, the agent refunded the wrong order and left a note saying “resolved.”

A language-model evaluation might give that answer a high score. It is fluent, relevant, and contains the word “refunded.” A useful agent evaluation asks a less flattering question: did the payment system end in the correct state?

That is the central shift. An agent evaluation, or agent eval, measures whether an agent completed a multi-step task in an environment. An environment is the world the agent can inspect and change: a code repository, browser, desktop, or internal API. Success is checked from the resulting state, not from whether the final sentence resembles a reference answer.

A benchmark is a shared set of tasks, an environment, and a scoring procedure. Benchmarks compare systems; your own eval suite tells you whether Tuesday’s prompt change broke what customers use.

The major agent benchmarks

Each benchmark probes a different kind of agency.

SWE-bench gives an agent a real issue from a real software repository. It must inspect code, edit files, and run tests. The harness applies the patch at the task’s base commit, then runs designated FAIL_TO_PASS regression tests and PASS_TO_PASS preservation tests. It does not require the entire repository suite to pass, so a missing regression test can let an incorrect patch through. SWE-bench Verified is a human-validated subset intended to remove ambiguous tasks.

The meaningful question is whether the repository behaves correctly after the change, not whether the patch resembles a reference patch. SWE-bench measures a demanding slice of coding work; it does not establish security, maintainability, or production-readiness.

GAIA tests general-assistant tasks combining reasoning, web research, file handling, and tools. Its grader generally checks an exact answer, making it useful when the answer itself is the product. It is not a stateful business-workflow test: an agent can answer correctly without changing any system.

WebArena puts an agent in self-hosted web applications and asks it to complete browser tasks. Evaluation checks backend records or another target state, such as an order, post, or setting. It tests navigation, information retention, and recovery from UI mistakes; a convincing screenshot is not enough.

OSWorld evaluates computer-use agents across desktop operating systems and real applications. Tasks include manipulating files, using office software, and changing settings. It exposes failures API tests hide: wrong coordinates, lost focus, misunderstood dialogs, and typing into the wrong window.

tau-bench evaluates tool-using agents in customer-service-style dialogues with simulated users and APIs, including policy compliance. AgentBench is a broader collection of environments. Choose the benchmark whose environment and failure costs resemble the risk you care about.

As of August 2026, public benchmarks are comparison signals and smoke tests, not usually release gates. Scores depend on model access, tool permissions, harness versions, task contamination, and retry rules. A private, domain-specific suite should decide whether your agent ships.

Why agent evaluation is noisy

Outcome verification needs infrastructure: a checked-out repository and test runner, a reset browser application, or a desktop VM. That cost buys evidence that the world changed correctly.

An agent also produces a trajectory: model messages, tool calls, observations, and results. Every decision is another chance to choose the wrong tool or misread an observation. Sampling and tool timing add randomness, so one run is an anecdote, not a measurement.

Pass@k

If an agent succeeds with probability p on each independent attempt, and at least one of k attempts counts as success:

pass@k = 1 - (1 - p)^k

For p = 0.40:

p = 0.40

for k in [1, 2, 3, 5, 10]:
    passk = 1 - (1 - p) ** k
    print(f"pass@{k:<2}: {passk:.1%}")

pass@1 : 40.0%
pass@2 : 64.0%
pass@3 : 78.4%
pass@5 : 92.2%
pass@10: 99.4%

The agent did not improve; the allowed attempts changed.

If each attempt costs 8 cents and takes 35 seconds, five attempts cost 40 cents and can take almost three minutes per task, before environment startup. For production actions, the larger cost may be a duplicate payment or deletion.

Track process metrics as well as outcomes:

• tool calls, tokens, latency, and cost;
• tool errors, invalid arguments, and recovery attempts;
• policy violations, unsafe actions, and human interventions.

An 80-percent agent using 60 tool calls is not equivalent to one using 12.

A worked eval: the duplicate refund

Consider this task:

Order A1842 contains two charges of 4,999 cents. Refund exactly one duplicate charge, add a support note with reason duplicate_charge, and change nothing on any other order.

The agent can inspect an order, create a refund, and append a support note. A strong evaluator checks exactly one new refund for A1842 and 4,999 cents, exactly one correctly associated note, unchanged non-target orders and protected state, and a safe tool sequence: inspect A1842 before refunding, then add the note.

A final “Done” is only a claim. The checker should inspect state and the recorded trajectory independently:

def state_is_correct(initial, final):
    old_refunds = {r["refund_id"]: r for r in initial["refunds"]}
    new_refunds = {r["refund_id"]: r for r in final["refunds"]}
    old_notes = {n["note_id"]: n for n in initial["notes"]}
    new_notes = {n["note_id"]: n for n in final["notes"]}

    if any(new_refunds.get(k) != v for k, v in old_refunds.items()):
        return False
    if any(new_notes.get(k) != v for k, v in old_notes.items()):
        return False

    added_refunds = [r for k, r in new_refunds.items() if k not in old_refunds]
    added_notes = [n for k, n in new_notes.items() if k not in old_notes]
    if len(added_refunds) != 1 or len(added_notes) != 1:
        return False

    refund, note = added_refunds[0], added_notes[0]
    if (refund["order_id"], refund["amount_cents"]) != ("A1842", 4999):
        return False
    if (note["order_id"], note["reason"]) != ("A1842", "duplicate_charge"):
        return False
    if final["orders"]["A1842"]["status"] != "refunded":
        return False

    for order_id, order in initial["orders"].items():
        if order_id != "A1842" and final["orders"].get(order_id) != order:
            return False
    return final["protected_state"] == initial["protected_state"]


def trajectory_is_safe(trajectory, allowed_tools):
    inspected = attempted = False

    for call in trajectory:
        tool = call["tool"]
        args = call["arguments"]
        if tool not in allowed_tools:
            return False

        if tool == "inspect_order" and args.get("order_id") == "A1842":
            inspected = True
        elif tool == "create_refund":
            if (not inspected or attempted or
                args.get("order_id") != "A1842"):
                return False
            attempted = True
        elif tool == "append_support_note":
            if not attempted or args.get("order_id") != "A1842":
                return False

    return True


def refund_is_correct(initial, final, trajectory, allowed_tools):
    return (
        state_is_correct(initial, final)
        and trajectory_is_safe(trajectory, allowed_tools)
    )


initial = {
    "refunds": [],
    "orders": {
        "A1842": {
            "status": "paid",
            "charges": [
                {"charge_id": "C1842-1", "amount_cents": 4999},
                {"charge_id": "C1842-2", "amount_cents": 4999},
            ],
        },
        "B7711": {"status": "paid"},
    },
    "notes": [],
    "protected_state": {"payouts_enabled": True, "account_flags": []},
}

final = {
    "refunds": [{
        "refund_id": "R9001",
        "charge_id": "C1842-2",
        "order_id": "A1842",
        "amount_cents": 4999,
    }],
    "orders": {
        "A1842": {
            "status": "refunded",
            "charges": [
                {"charge_id": "C1842-1", "amount_cents": 4999},
                {"charge_id": "C1842-2", "amount_cents": 4999},
            ],
        },
        "B7711": {"status": "paid"},
    },
    "notes": [{
        "note_id": "N5001",
        "order_id": "A1842",
        "reason": "duplicate_charge",
    }],
    "protected_state": {"payouts_enabled": True, "account_flags": []},
}

trajectory = [
    {"tool": "inspect_order", "arguments": {"order_id": "A1842"}},
    {"tool": "create_refund", "arguments": {
        "order_id": "A1842", "charge_id": "C1842-2", "amount_cents": 4999,
    }},
    {"tool": "append_support_note", "arguments": {
        "order_id": "A1842", "reason": "duplicate_charge",
    }},
]

allowed_tools = {"inspect_order", "create_refund", "append_support_note"}
print(refund_is_correct(initial, final, trajectory, allowed_tools))

True

The state checker compares stable records and protected snapshots; the trajectory checker does not trust the agent’s claim about what it changed.

For a disallowed fixture, declare expected_outcome = "refuse_refund". Require a refusal, zero new refunds, zero notes, and an unchanged environment, while still checking the allowed-tool policy. Correct refusal must not be sent through the positive refund checker.

Now run a suite of, for example, 20 cases varying order, amount, customer history, and policy. Run each case once with five fixed seeds. If 15 of 20 pass on the first attempt, pass@1 is 75 percent. If 19 pass at least once across five attempts, pass@5 is 95 percent. These answer different questions: first-try reliability versus eventual success in a safe sandbox. Reset the environment before every episode, or later attempts may inherit earlier refunds and measure state leakage.

Eval-driven development

Eval-driven development makes evaluation the engineering loop: define representative tasks, run them after changes, inspect failures, and use the evidence to choose the next change.

1. Build representative tasks

Start with production logs, support tickets, incidents, and edge cases. Remove secrets and personal data. Each task needs:

• a natural-language request and known starting fixture;
• explicit tools and permissions;
• a programmatic success condition.

Include risky negative cases: a partially paid order, currency mismatch, legal hold, or request to refund another customer. Write the contract before inspecting outputs so the goalposts do not move.

2. Make episodes reproducible

Use isolated databases, disposable repositories, browser profiles, or VMs. Pin the fixture and record model, prompt, tool definitions, seed, timeout, and budget. Preserve tool arguments, results, timestamps, retries, and the final snapshot; redact secrets. Reset before every episode and set time and tool-call limits.

3. Keep the checker independent

Inspect facts the agent could not simply write into existence: the ledger for a refund, tests in a clean checkout, or a setting read through an independent path. Verify side effects directly, not just final text.

A language-model judge can help with fuzzy work such as summaries if it has a written rubric, examples, and human calibration. Do not use an uncalibrated judge to approve transfers, deletions, or access-policy changes.

4. Report useful evidence

An episode is one attempt at one task. Report task count, seeds, pass@1 and pass@k, cost, latency, and per-task results. Equal-weight tasks before averaging; otherwise many easy duplicates can hide failures.

For 75 successes in 100 independent episodes, the observed rate is 75 percent and the approximate 95-percent interval is about ±8.4 percentage points. Small differences may not be improvements. Flaky environments and correlated failures make that approximation less reliable, so report infrastructure failures separately.

5. Diagnose and gate releases

Label failures such as wrong tool, invalid argument, missing inspection, misunderstood observation, loop, premature stop, policy violation, or budget overrun. Eight failures beginning with the wrong customer lookup point toward a tool schema or state-exposure problem, not necessarily a prompt problem.

Run a smoke subset on every pull request and the full suite after model, prompt, tool-schema, routing, or infrastructure changes. Keep fixed seeds for regression detection and add fresh seeds periodically. Gate on safety, pass@1, p95 latency, cost, and tolerated regression—not pass rate alone.

Failure modes you will actually see

Pass@5 rises but pass@1 does not. You changed retries, not capability. Report each k, plus the time, cost, and side effects required.

Runs disagree. Check environment reset, external-service flakiness, and seeds. Report infrastructure failures separately from agent failures.

The dashboard is green but users complain. The verifier is weak. It may check status = refunded without checking amount, owner, permissions, or collateral changes. Strengthen end-state assertions and add negative cases.

The agent passes but behaves dangerously. Outcome-only scoring hides unsafe paths. Add trajectory and policy checks; make critical violations hard failures. See agent safety controls and agent reliability.

When not to use a benchmark score

Do not use a public benchmark as the only launch criterion for a domain with different tools, policies, or failure costs. A browser benchmark cannot certify banking, and a coding benchmark cannot certify deployment.

Do not optimize for pass@k on irreversible actions unless attempts are isolated or tools are idempotent. Idempotent means repeating a request produces the same final effect rather than performing it twice. Reads are usually safe to retry; refunds, deletions, and emails may not be.

Use public benchmarks for broad capability, private tests for regressions, and human review for ambiguous or high-impact cases. No single score does all three jobs.

In one breath

• Agent evals verify a multi-step task’s outcome in an environment, not just its final text.
• SWE-bench tests repository fixes; GAIA tool-assisted answers; WebArena web state; OSWorld desktop tasks; and tau-bench policy-aware tool dialogues.
• Report pass@1 and pass@k with seeds, retries, task counts, cost, and latency.
• Reset environments, retain trajectories, use independent checkers, and measure process as well as outcome.
• Public benchmarks are signals; your regression suite tells you whether the 3 a.m. refund will be correct.

Quick check

Next

Evaluating agents covers trajectory-scoring methods and judge design in more detail. Agent observability shows how to retain the traces you need to diagnose failures, while agent reliability covers safe side effects and recovery in production.