Observability & tracing

At 3:07 a.m., a customer asks an agent to cancel an order. The agent eventually says, “I could not complete that.” The customer sees one sentence. Your team sees a support ticket, a latency alert, and a cloud bill that is 40 percent higher than yesterday.

The cause might be a slow payment API. Or a model that selected the wrong tool. Or a tool that failed twice and was silently retried. Or a prompt that grew from 4,000 tokens to 40,000 because the agent kept adding conversation history. The final answer does not tell you which one happened.

You need the run itself, not just its conclusion. That is observability: the ability to understand what a running system did from the data it emits. For agents, the most useful piece of that data is a trace, a connected record of one request from its starting point to its final result.

Monitoring tells you that; tracing tells you where

Monitoring aggregates measurements across many requests. It answers questions such as:

• Did the error rate rise?
• Is p95 latency above its target?
• Are tool failures or cost per run increasing?

Monitoring detects fleet-level problems. Tracing follows one request through its internal operations: which model call was slow, which retrieval query returned nothing, which tool received the wrong argument, and which retry used the time.

The practical loop is: monitoring says the agent became slow; a representative trace identifies the failing operation; monitoring confirms whether that pattern is widespread.

A log is a timestamped event such as “payment timeout.” Logs are useful for searching exact errors, but a trace preserves the relationships among nested calls. An evaluation asks whether the result was good—whether the agent cancelled the correct order and followed policy. A trace asks what happened while it tried.

The mental model: a run made of spans

An agent run is usually a tree of operations. The root might be agent.run, with child spans for model calls, retrieval, tools, retries, approvals, and the final response.

A span is one timed operation with a start time, end time, name, status, and structured attributes such as model, tool, token count, or HTTP status. A trace is the connected collection of spans for one run. A trace ID finds all related spans.

The root gives total duration; child spans show where time and usage went. The nesting shows execution relationships and timing, helping you infer likely contributors; it does not prove causation.

A child span carries a parent span ID. When work crosses an HTTP, queue, or service boundary, trace context must be propagated. Without it, the downstream service creates an unrelated trace and the viewer has a hole at the most useful point.

Parallel work changes latency arithmetic. Two calls taking 800 milliseconds can add about 800 milliseconds to wall-clock latency when they overlap, not 1.6 seconds. Their token and dollar costs still add together. Most traces are trees, though asynchronous jobs and messages between agents may require links between related spans.

A trace viewer should let you expand spans and filter by model, tool, status, user, deployment version, and time range. A waterfall is useful; searchable, correlated data is what gets you through an incident.

A worked trace: where did 4.8 seconds and 7 cents go?

Consider a support agent handling:

“Cancel order 1842 and refund it if it has not shipped.”

The agent has a policy retriever, an order lookup tool, and a refund tool. An illustrative trace contains:

The three refund attempts take 2.4 seconds. If the first had succeeded, this run would have been about 1.6 seconds faster, assuming the rest stayed the same. That is a better diagnosis than “the model is slow.”

The two displayed model calls account for 5.1k output tokens, not the root’s 12.4k total model tokens. The root total includes input and output across every model call, including system prompts, tool schemas, retrieved text, history, and any omitted calls.

Track reasoning tokens according to the provider’s accounting without adding them twice. Keep embedding or other billable work in a separately labelled total.

The 7 cents is an estimate based on the model’s input, output, cached-input, and possibly reasoning-token rates. Calculate it with the price-card version active when the run happened and store that version with the trace. Otherwise, a price change can look like an agent regression.

Inspect the decisions as well as the timings:

• Did the model check shipment status before selecting tool.refund?
• Did it pass validated {"order_id": "1842"}?
• Did policy retrieval return the right document?
• Were timeouts classified as retryable?

Retrying a read is usually harmless. Retrying a charge or refund can duplicate a side effect unless the payment service supports an idempotency key. Tracing shows the three attempts; it cannot make an unsafe retry safe.

What to instrument

At the request boundary, extract incoming trace context and create a server span. It is a new root only when no valid parent exists. Attach bounded dimensions such as environment, endpoint, agent version, and model-policy version. Keep raw tenant IDs access-controlled; use a bounded tenant tier or group for metrics.

Create child spans around operations that affect correctness, latency, cost, or safety:

• Model calls: provider, model, prompt version, input/output tokens, reasoning usage when supplied, latency, finish status, and selected tools.
• Tools: name, validated argument shape, result status, timeout, retry number, downstream service, and request or idempotency ID.
• Retrieval: source, query type, result count, filtering policy, latency, and document IDs.
• Control flow and guardrails: plans, handoffs, approvals, loop iterations, step-limit termination, and checks that block or transform data.
• Remote services: propagated context, service name, operation, and status.

Record success and failure. “Tool call completed” must distinguish a valid empty result from a timeout converted into an empty result.

Capture prompts and payloads only when policy permits. Redact secrets and personal data before export. For sensitive values, a type, length, or access-controlled internal ID may be enough. If stable correlation requires a transformed value, use a keyed HMAC or approved tokenization rather than an ordinary hash: emails, order IDs, and phone numbers are dictionary-matchable.

Propagate trace context through HTTP calls and queues. A nested consumer can be a child span; independently processed work can start a new trace with a span link. Keep a human-facing request ID as well.

Export asynchronously through a bounded buffer, retain errors when full, and make telemetry failure non-fatal. The agent should not fail a refund because its telemetry endpoint is unavailable.

Reading and operating traces

When a run is wrong or slow:

1. Start at the root: duration, status, step count, versions, and termination reason.
2. Follow the critical path. A large parallel child may not determine latency.
3. Inspect failures, retries, empty retrievals, validation errors, and loops.
4. Compare the model’s decision with the tool argument and result; then check input-token growth, prompt versions, and model changes.

For latency, the critical path matters. For cost, all billable parallel work matters. For correctness, find the first divergent decision, not just the final bad sentence. Replay only safe recordings; do not casually repeat a production refund or purchase.

At the analysis layer, dashboard fleet symptoms and alert on timeout rate, step-limit terminations, sudden input-token growth, and cost per successful task. Keep bounded dimensions such as model version, prompt version, tool, and error class.

Randomly sample routine successes, but retain errors, timeouts, policy blocks, and latency or cost threshold breaches. Tail-based sampling can retain rare bad runs by deciding after completion. Set retention by use case and do not keep raw model outputs indefinitely.

Failure modes you will actually see

One giant span. A total duration with no children is a stopwatch, not an explanation. Instrument model, retrieval, tool, retry, and control-flow boundaries separately.

A gap at a service. The downstream service created a second trace because context propagation was missing across HTTP, a queue, or a worker. Test propagation in integration tests and retain the service request ID.

Hidden retries. A tool that normally takes 800 milliseconds appears to take 2.4 seconds. Create a span for each attempt with retry number, backoff, error class, and final result. Check idempotency before increasing retries.

Success but wrong answer. A successful tool response does not prove that the model interpreted it correctly. Record document IDs, tool status, model version, and final output, then send the case to evaluation.

Missing or expensive traces. Random sampling can discard rare failures, and high-cardinality prompts, URLs, or user IDs can make the backend costly. Retain threshold breaches, use tail sampling when appropriate, keep unbounded values as searchable fields rather than metric dimensions, and cap payload sizes.

The honest limit

Tracing is not a recording of the model’s mind. It shows the prompt sent, tool selected, output returned, and timing around them. It does not prove hidden reasoning was sound. Private chain-of-thought is not a required observability field.

It also cannot reveal work you failed to instrument. Prompt assembly, caching, retries, or safety checks outside your spans can make a run look simpler than it was. Test instrumentation by forcing a timeout, retry, and parallel branch, then verify the expected trace shape.

Use traces for “what happened here?”, monitoring for “is this happening across the fleet?”, evaluations for “was the behaviour good?”, and audit and authorization records for “was this action allowed?”

Quick check

Next

Once a trace shows where the time and tokens go, the next question is what to change without making the agent less useful. That is the bridge to cost and latency control.