Evaluating agents

At 3:07 a.m., a support agent tells a customer, “Your 49-dollar refund has been issued.”

The customer is relieved. The payment ledger says nothing happened.

A second run finds the order, checks the policy twice, and issues the refund twice. The final message still looks perfect. Your screenshot-based test passes. Your finance team does not.

This is the problem agent evaluation exists to solve. An agent is a model-driven program that chooses its next step from the information it has observed.

Its answer is only one part of the execution. It may search, call APIs, retry, change state, and send messages along the way.

A single LLM call has one input and one output. An agent has a trajectory, the ordered record of its model calls, tool calls, observations, and decisions. It also has side effects, changes outside the model such as issuing a refund, modifying a ticket, or sending an email.

Two runs of the same task can take different paths and both be correct. They can also produce the same polished answer while one path leaks data or makes a dangerous change. “It worked when I tried it” is one anecdote, not an evaluation.

What exactly gets scored?

There are three useful questions:

1. Did the user receive the required result or response? This is the outcome.
2. Did the agent behave acceptably? This is the trajectory: its actions, arguments, order, budget, and safety constraints.
3. Did the outside world end up right? This is the state assertion: whether the database, payment system, ticket, or message history contains the intended result.

Outcome checks judge what the user sees. State assertions verify that external mutations actually occurred as intended.

A response can have the expected format while the payment system fails to apply the change or applies it twice. A tool returning {"status": "ok"} is evidence, not proof, unless you verify the resulting state.

For the refund agent, an outcome check asks whether the final response explains the policy and communicates the decision correctly. A trajectory check asks whether it called lookup_order before issue_refund, used an allowed tool, supplied the right order identifier, and stopped within a reasonable number of steps.

A state check asks whether exactly one 49-dollar refund exists for that order.

The mechanism is straightforward. The model selects actions from a changing context. Each action changes the next context, and some actions change the real world. A bad intermediate action can therefore be harmful even when a later action repairs the conversation. Final text cannot reveal all of that.

Do not demand one exact “golden trajectory” for every task. If the agent can verify an order through either an internal lookup or a trusted cache, both paths may be valid. Score invariants instead: conditions that must always hold.

“The customer is verified before a refund” is an invariant. “Step 1 must be tool A” is usually an implementation detail.

The evaluation loop

A useful production pattern is:

1. Capture a complete trace.
2. Check each action and the resulting state.
3. Score the final outcome.
4. Aggregate results by task, version, and failure type.
5. Block or investigate changes that cross a pre-agreed threshold.

Scoring only the final answer loses the evidence needed to explain a failure; scoring only tool calls can miss an incorrect response. The trace connects the two.

A trace should normally contain the task identifier, agent and model version, system prompt version, tool schemas, every tool name and argument, tool result, timestamps, token and cost data, errors, retries, and final state assertions.

Redact credentials and unnecessary personal data. A trace full of secrets is not observability; it is a future incident report.

A worked score with real numbers

Suppose you have 50 refund tasks and run each one four times because the model and external services are not perfectly deterministic. That produces 200 runs.

The candidate version completes 186 runs correctly. Its outcome rate is 186 divided by 200, or 93 percent. Fourteen runs failed the task outcome.

Among those 200 traces:

• three used an unauthorized tool;
• nine exceeded a six-step budget;
• two passed malformed order identifiers.

These categories can overlap: one run might both use an unauthorized tool and exceed the budget. They explain why runs failed; they are not supposed to add up to the number of failed outcomes.

Suppose the release policy says:

• outcome success must be at least 95 percent;
• unauthorized tools must occur zero times;
• the 95th-percentile latency, meaning the time no more than 5 percent of runs exceed, must stay below 8 seconds.

The candidate fails immediately on outcome and unauthorized-tool use. A new version that succeeds on 194 of 200 runs reaches 97 percent, but it still fails if one trace uses an unauthorized tool. Averages do not wash away safety violations.

Keep aggregate and per-task results. An overall 97 percent can hide one customer segment at 80 percent if the dataset contains too many easy cases. Group by language, product tier, tool, policy type, and adversarial condition.

Here is a deliberately small trajectory scorer. It demonstrates the separation; it is not a complete safety system.

allowed_tools = {"lookup_order", "get_policy", "issue_refund", "escalate"}
step_budget = 6

def report(label, trajectory, task_succeeded):
    valid_args = all(step["args_valid"] for step in trajectory)
    in_scope = all(step["tool"] in allowed_tools for step in trajectory)
    within_budget = len(trajectory) <= step_budget

    passed = task_succeeded and valid_args and in_scope and within_budget
    verdict = "PASS" if passed else "FAIL"

    print(
        f"{label}: succeeded={task_succeeded}  "
        f"in_scope={in_scope}  within_budget={within_budget}  "
        f"steps={len(trajectory)}  -> {verdict}"
    )

clean = [
    {"tool": "lookup_order", "args_valid": True},
    {"tool": "get_policy", "args_valid": True},
    {"tool": "issue_refund", "args_valid": True},
]

messy = [
    {"tool": "lookup_order", "args_valid": True},
    {"tool": "web_search", "args_valid": True},
    {"tool": "get_policy", "args_valid": True},
    {"tool": "get_policy", "args_valid": True},
    {"tool": "lookup_order", "args_valid": True},
    {"tool": "get_policy", "args_valid": True},
    {"tool": "issue_refund", "args_valid": True},
]

report("clean", clean, task_succeeded=True)
report("messy", messy, task_succeeded=True)

clean: succeeded=True  in_scope=True  within_budget=True  steps=3  -> PASS
messy: succeeded=True  in_scope=False  within_budget=False  steps=7  -> FAIL

Both runs have task_succeeded=True, so an outcome-only check would pass both. The trajectory check catches the extra tool and seventh step.

It does not check whether the order ID was the customer’s order, the refund amount was 49 dollars, or the payment ledger changed exactly once. Those require semantic argument checks and external state assertions.

Build an eval set that teaches you something

An eval set is a versioned collection of tasks, fixtures, expected conditions, and scoring rules. Start with real work. Every production incident should become a regression case after sensitive data is removed.

For each refund case, store:

• the customer request and relevant initial state;
• allowed tools and safety constraints;
• the required user-facing outcome and separate state condition, such as an accurate response plus one refund for the correct order and amount;
• acceptable alternatives, such as escalation when the policy is unclear;
• expected failure behavior, including refusal or human handoff;
• metadata such as locale, order age, payment method, and whether the request is adversarial.

Include ordinary and unpleasant cases:

• a valid refund;
• an expired policy window;
• similar order identifiers;
• missing identity;
• a request involving another person’s order;
• a tool timeout;
• a duplicate retry;
• prompt injection inside an order note.

Happy-path demos are poor detectives.

Run tasks in a sandbox or against reversible fixtures. For issue_refund, use a fake payment service that records calls and supports failure injection. Assert the resulting ledger, not just the tool response.

If the integration cannot be safely simulated, put a human approval gate before the mutation. Never discover whether the evaluator works by refunding a real customer.

A task may have several valid trajectories. Check that:

• identity verification happened before the mutation;
• the order ID came from a trusted lookup;
• the refund amount did not exceed the eligible amount;
• only approved tools were used;
• retries did not create more than one mutation;
• the final response accurately reported the resulting state.

This is more durable than recording one exact list of tool names. Exact traces are appropriate when order is itself the requirement, such as “confirm authorization before transferring money.” They are brittle when harmless alternatives are allowed.

Repeat stochastic tasks. One successful run demonstrates only observed capability on that attempt. Repeated failures indicate poor observed reliability, but five runs of one fixture are not five independent task cases.

Run more times, aggregate at the task level, and report confidence or bootstrap intervals rather than treating a small percentage as settled.

Record individual runs and the model, provider, prompt, tool schema, retrieval index, and framework versions that created them. Otherwise a red result tells you that something changed but not what.

Deterministic checks first, judges second

Use a deterministic evaluator, a rule whose result does not depend on another model, wherever the requirement is mechanical:

• compare database state before and after;
• validate tool arguments against a schema;
• compare requested and actual order IDs;
• count mutations;
• check an allowlist;
• measure steps, latency, tokens, and cost.

These checks are cheap, explainable, and stable. They should decide safety properties. An LLM judge should not wave through an unauthorized transfer because the explanation sounded responsible.

An LLM-as-judge is another model that scores a response or trace against a rubric. It is useful for open-ended qualities such as whether a policy explanation is complete, a research answer well-supported, or a handoff message clear.

It is not ground truth: judges can prefer longer answers, be swayed by confident language, and change when the judge model changes.

Give the judge a narrow rubric with observable evidence. Ask for structured fields such as correct, grounded, and reason, with citations to the trace or reference material.

Calibrate it against a human-labeled sample and version the rubric and judge model. For high-stakes actions, use a human or deterministic policy gate for authorization. A judge can assess whether a response sounds appropriate; it should not be the final authority to move money.

Choose the evaluation method by the risk

The deciding axis is whether expected behavior is mechanically knowable and how costly a wrong action is.

Failure modes you will see first

The dashboard is green, but customers report wrong actions.
A high final-answer score alongside incorrect database or payment records means the evaluator checked text and tool syntax but not external state. Fix it with before-and-after assertions, including mutation counts and exact identifiers.

Every harmless variation is reported as a regression.
A model changes wording or chooses a different read-only tool, and failures spike because the eval compares the candidate with one golden sequence. Replace sequence equality with invariants and permitted alternatives. Keep exact ordering only where policy requires it.

The agent passes offline and fails in production.
Friendly fixtures hid tool timeouts, permission errors, stale data, or schema mismatches. Add contract tests, realistic error injection, permission boundaries, rate-limit responses, and controlled integration runs.

A retry creates two side effects.
After a timeout, the model calls the mutation again, creating duplicate refunds, tickets, or messages. Make mutations idempotent with a stable request key, assert at-most-once state, and test ambiguous tool responses explicitly.

The average looks healthy while a subgroup is broken.
Break down results by locale, product tier, tool, and other meaningful slices. A hundred easy cases can hide ten disastrous ones.

CI is where evaluation becomes engineering

Run a small smoke set on every pull request and a broader set on scheduled or release builds. Compare the candidate with a known baseline, but keep separate gates for outcome quality, safety violations, state correctness, latency, and cost.

A useful report says:

• 194 of 200 runs achieved the intended outcome;
• 2 runs supplied invalid tool arguments;
• 0 runs used a forbidden tool;
• median latency was 2.1 seconds and p95 latency was 7.4 seconds;
• failures occurred in policy-window and duplicate-retry cases.

That gives an engineer somewhere to look. “Quality: 97 percent” does not.

When a check fails, save the trace and smallest reproducible task. Inspect the first divergence from acceptable behavior, not merely the final error.

The first wrong order ID, unexpected tool, or missing observation often explains all later steps.

Quick check

Next

You cannot evaluate what you cannot see. Observability and tracing covers the traces these checks consume, while cost control keeps repeated evaluation and production runs affordable.