datarekha
MLOps Hard

How do you evaluate an agentic system, and what is the difference between trajectory and outcome evaluation?

For AI / LLM EngineerMLOps Engineerresearch-engineer
The short answer

Outcome evaluation checks whether the agent's final result is correct, while trajectory evaluation inspects the intermediate steps, tool calls, and decisions along the way. You need both because an agent can reach the right answer through a flawed path or fail despite sound reasoning; trajectory metrics catch wrong tool use, redundant steps, and loops that outcome-only metrics miss.

How to think about it

Outcome evaluation checks whether the agent’s final result is correct, while trajectory evaluation inspects the intermediate steps, tool calls, and decisions along the way. You need both because an agent can reach the right answer through a flawed path or fail despite sound reasoning; trajectory metrics catch wrong tool use, redundant steps, and loops that outcome-only metrics miss.

Learn it properly Evaluating agents

Keep practising

What is an AI agent, and how does it differ from a single LLM call? Explain the ReAct agent pattern and how it compares to Plan-and-Execute and Reflexion. How do you evaluate the quality of an LLM or RAG system? What are the key regression metrics — MAE, RMSE, MAPE, R² — and what are their failure modes? When should you use a multi-agent system versus a single agent, and what is the supervisor versus swarm pattern?
All MLOps questions

Explore further

Reflection Agentic design patterns AGENTS.md, Skills & Tools
Agent Trajectory Agent Reflexion ReAct
Skip to content