How do you evaluate an agentic system, and what is the difference between trajectory and outcome evaluation?

How to think about it

An agentic system needs both outcome evaluation and trajectory evaluation. Outcome evaluation asks whether the task ended correctly and safely; trajectory evaluation asks whether the sequence of observations, tool calls, arguments, and decisions was valid, efficient, and policy-compliant.

Why outcome alone is not enough

Imagine a support agent handling this request:

“Refund the duplicate charge on order 8472. Do not touch my annual subscription.”

The final text might say, “Your $120 duplicate charge has been refunded.” An outcome evaluator that reads only that sentence may mark the task as a success.

That is not enough. The agent might have refunded the subscription instead, issued two refunds, or skipped the policy check. The prose can be perfect while the customer’s bank account is very much not.

Outcome evaluation checks the endpoint. The endpoint must include more than the agent’s final message:

• Did it choose the correct order?
• Was the refund amount exactly $120?
• Did the payment system record one refund rather than two?
• Did it avoid the protected subscription?
• Did the user receive an accurate explanation?
• Did the agent complete the task within the allowed time and cost?

For a purely informational agent, the endpoint may be a text answer. For an agent that changes records, sends email, moves money, or deploys code, the endpoint includes the resulting external state and side effects.

Trajectory evaluation checks the route taken. A trajectory is the time-ordered record of what the agent observed, which actions it selected, what arguments it sent, what tools returned, and how it updated its plan.

The route matters because two agents can produce the same final answer while having very different safety profiles. One followed the authorization rules. The other got lucky.

Common misconception: A trajectory is not the model’s private chain of thought. I evaluate observable events and state transitions: tool selection, arguments, results, retries, decisions, and stopping behavior. I do not need to collect or score hidden reasoning word for word.

A concrete example

Suppose the support agent has four hypothetical tools:

• lookup_order
• get_refund_policy
• issue_refund
• send_email

A healthy trace might look like this:

lookup_order(order_id=8472)
  -> amount=$120, type=duplicate, status=charged

get_refund_policy(customer_id=391)
  -> duplicate charges refundable, subscription excluded

issue_refund(order_id=8472, amount=$120)
  -> refund_id=R-551, status=accepted

send_email(customer_id=391, refund_id=R-551)
  -> status=sent

The outcome grader should verify the payment record, not merely trust the last message. It can check that refund R-551 exists, belongs to order 8472, has amount $120, and appears only once.

The trajectory grader can check different properties:

• The agent looked up the requested order before mutating payment state.
• It retrieved the refund policy before calling the refund tool.
• It passed the correct order identifier and amount.
• It did not call the subscription tool.
• It stopped after the email succeeded.
• It did not retry a successful refund and risk a duplicate charge.

Now consider three traces.

In the first, the agent calls issue_refund twice because the first response is slow. The payment service is not idempotent, so the customer receives $240. If the final message still says “Your $120 refund is complete,” a text-only outcome evaluator passes while the real outcome fails. A trajectory evaluator catches the duplicate mutation, and a state-based outcome check catches the over-refund.

In the second, the agent follows the correct sequence but the payment service times out after accepting the request. The trajectory looks reasonable, but the task may be incomplete or ambiguous. This is why evaluation must distinguish agent error from infrastructure failure. The right production response may be to query refund status, not blindly retry.

In the third, the agent calls the refund tool before reading policy, but the tool rejects the request and the agent later succeeds. The final state is correct. Outcome evaluation may pass. Trajectory evaluation should still record a policy-order violation because the same behavior could be dangerous when a tool does not enforce that rule.

How I would build the evaluation

I would start with a task contract. For the refund example, the contract states the permitted order, maximum amount, required authorization, prohibited resources, and acceptable final states. Without that contract, “good trajectory” becomes an aesthetic opinion.

Then I would create a test set with realistic variation:

• duplicate charges and legitimate charges
• missing order identifiers
• customers with several recent orders
• expired or conflicting policies
• tool timeouts and malformed responses
• prompt-injection text inside an order note
• repeated user requests

The test set should contain expected outcomes and the constraints that every valid trajectory must obey. I would also divide results into slices. An overall success rate can hide the fact that the agent fails almost every time a customer has two orders or a tool returns a partial response.

I would use the strongest grader available for each property:

Property	Example check	Preferred grader
External state	One refund of exactly $120 exists	Database or API assertion
Tool arguments	Correct order ID and amount	Deterministic trace rule
Policy compliance	Policy read before refund	Trace rule or state machine
Final explanation	Clear and factually accurate email	Rubric-based human or model judge
Efficiency	No unnecessary calls or loops	Trace metrics
Safety	No unauthorized mutation	Hard-fail rule and human review

Deterministic checks should come first because they are reproducible. If the database says the agent refunded the wrong order, a language model should not be allowed to overrule it with a generous interpretation.

For open-ended output, I can use a model-based judge, but I would give it a narrow rubric: factual accuracy, completeness, tone, and whether the answer reflects the verified state. I would calibrate it against human labels and monitor agreement. Model judges can be inconsistent, prefer verbose answers, or miss subtle policy violations. They are useful graders, not magical truth machines.

For trajectory metrics, I would report several measures rather than inventing one grand score:

• task success rate
• policy-violation rate
• invalid tool-call rate
• unnecessary-call rate
• loop or retry rate
• median and ninety-fifth-percentile number of steps
• latency and token cost
• recovery rate after tool failure

For example, on 1,000 refund tasks, I might report 930 successful end states, 18 traces with a prohibited call, 31 traces that exceeded the step budget, and a ninety-fifth-percentile path length of 9 calls. These categories can overlap. That is useful information; collapsing everything into “93 percent accuracy” is not.

The nuance that earns the senior signal

There is rarely one correct trajectory. An agent might read the policy before or after looking up the order. It might use a cached policy or call the policy service directly. If both paths are safe and produce the same authorized result, penalizing one for not matching a recorded “golden trace” would be a mistake.

I therefore score invariants and preconditions rather than exact sequences. A mutation must have authorization. A refund must target the requested order. A retry after an uncertain payment response must first establish whether the original mutation succeeded. Several different paths can satisfy those rules.

The reverse is also important: a valid-looking trajectory does not guarantee a good outcome. The agent can select the right tools and arguments but receive stale data, a poisoned retrieval result, or a faulty tool response. Evaluation should identify whether the failure came from planning, tool use, data quality, or infrastructure.

I would run offline evaluations in a sandbox before allowing side effects. For production systems, I would add approval gates for high-impact actions, shadow traffic where possible, and alerts on unusual trajectory patterns. Some outcomes are delayed. A refund may be accepted immediately but settle later, so the evaluator should distinguish “request accepted” from “money returned” and measure both.

The main trade-off is observability versus cost and privacy. Full traces are invaluable for debugging, but they can contain customer data and increase storage costs. I would redact sensitive fields, retain structured events longer than raw prompts, and restrict access. Logging everything without a retention policy is not observability; it is a future incident report.

A failure mode I would watch for

The first symptom is often a stable success rate paired with a rising cost and latency curve. Inspecting traces reveals that the agent repeatedly calls the same lookup tool after receiving a valid result, usually because the orchestration layer fails to mark the observation as consumed.

Outcome-only evaluation misses this when the final answer remains correct. Trajectory metrics expose the repeated calls, and a step budget limits the damage. I would then fix the state transition, add a regression test for the exact trace, and make mutating tools idempotent where possible so a timeout does not become a duplicate action.

What they’ll ask next

How do you evaluate an agent when several trajectories are valid?

I define required invariants, tool preconditions, and forbidden actions instead of comparing against one exact trace. I allow equivalent paths as long as they produce the required state and respect the safety constraints.

Should an LLM judge the trajectory?

It can help classify open-ended decisions, such as whether a plan was reasonable or an explanation was complete. I would not use it as the sole judge for payments, permissions, or safety-critical actions. Those need deterministic assertions, hard constraints, and sampled human review.

What would you put on the production dashboard?

I would show task success, policy violations, invalid or duplicate mutations, p95 latency, cost per task, step-count distribution, and results by important slices. A single success number hides too much. A system that succeeds 99 percent of the time but violates policy in the remaining 1 percent may still be unacceptable.

Say this in the interview

“I evaluate both the endpoint and the path: outcome checks whether the agent achieved the task and left the system in the correct state, while trajectory evaluation checks its tool calls, decisions, constraints, efficiency, and recovery behavior; deterministic checks handle side effects, and rubric-based evaluation handles open-ended quality.”