A production agent is slow and occasionally takes an unsafe action. What would you capture in traces and metrics across model calls, retrieval, tool execution, state transitions, and approvals so that you can diagnose both problems without logging sensitive content indiscriminately?

How to think about it

I would give every agent run one trace ID and record timed spans for model calls, retrieval, tool execution, state transitions, and approvals, with structured safety and latency metrics alongside them. I would log metadata, versions, sizes, hashes, decisions, and outcomes by default, while keeping raw prompts, retrieved text, and sensitive tool arguments out of ordinary traces.

Why this works

A trace is the end-to-end record of one agent run. A span is one timed operation inside it: a model call, database lookup, or payment attempt. Together, they answer the question a single duration cannot: where did the time go, and what decision led to the side effect?

The agent’s total latency is usually a sum of different waits:

• time queued before a model call
• time to the first model token
• time generating the remaining tokens
• retrieval and reranking
• tool network latency
• approval waiting time
• retries, backoffs, and state-machine loops

A metric might tell you that agent_run_duration p95 rose from 6 seconds to 11 seconds. A trace can tell you that the model stayed at 2 seconds, while a reranker began taking 4 seconds and human approvals added another 3 seconds.

The same causal view matters for safety. “The agent made an unsafe call” is not a diagnosis. The cause might be a bad model decision, stale retrieval, a schema or units mismatch, a policy evaluated against one action while the tool executed another, or an approval that expired but was still accepted.

Capture enough identifiers and decisions to distinguish those cases without copying the customer’s private conversation into every telemetry backend.

What the trace should contain

Use OpenTelemetry spans if that is already your platform’s tracing standard. The exact span names are less important than consistent parent-child relationships and stable attributes.

Layer	Capture in the span	Useful metrics
Agent run	agent version, workflow version, tenant class, start and end time, outcome, trace ID	total duration, success, failure, cancellation
Model call	provider, model deployment, region, prompt-template version, token counts, queue time, time to first token, completion time, stop reason, retry and fallback status	input and output tokens, error rate, timeout rate, cache-hit rate
Retrieval	index or collection version, embedding model version, query fingerprint, candidate count, top-k, score summaries, reranker version, cache status	retrieval latency, empty-result rate, reranker latency, freshness
Tool execution	tool name and version, argument schema version, validation result, authorization result, side-effect class, timeout, retry count, external request ID, outcome	call rate, duration, timeout rate, denied calls, blocked and completed side effects
State transition	state name, next state, transition reason, workflow version, loop count, wait duration	transition counts, loop rate, stuck-run rate, fallback rate
Approval	policy version, risk tier, canonical action hash, decision, approval latency, expiry, actor class, mismatch result	approval rate, denial rate, wait time, expired approvals, policy violations

A canonical action hash is a fingerprint of a normalized action. For example, the system should normalize currency, units, field ordering, and defaults before hashing an action. A keyed hash such as HMAC is safer than an ordinary hash for low-entropy values, because an attacker cannot cheaply guess the original value without the secret key.

For model calls, token counts are valuable because a sudden rise in input tokens often explains a slow run before anyone blames the model. Record the prompt-template ID and model configuration, not necessarily the prompt. If the model returns a structured tool call, record whether it passed schema validation, which fields were accepted, and a fingerprint of the output. Do not treat hidden chain-of-thought as an observability requirement. It is neither a reliable safety record nor a good reason to replicate private text across systems.

For retrieval, document IDs may themselves reveal sensitive information. Use an HMAC fingerprint or an internal opaque ID, and record document version, age, score distribution, and result count. Those fields can show that the agent used an old policy or received zero relevant documents without storing the policy text in the trace.

For tools, record the intended side-effect class: read-only, reversible write, financial write, credential change, or other high-impact action. Record the authorization and validation results separately. “The tool returned HTTP 200” is not the same as “the action was authorized and safe.” If the tool talks to an external system, capture its request ID and idempotency key so an operator can reconcile the trace with the external audit log.

State transitions deserve their own spans or events. An agent that silently cycles through plan, retrieve, and replan five times is slow for a different reason than one waiting on a human. Record state names and reason codes, not the entire state object.

Approvals are security boundaries, not just another model span. Bind the approval to the canonical action hash, policy version, actor or actor class, expiry, and exact tool version. Recheck authorization immediately before the side effect. The trace should show whether the action approved was the action executed.

A concrete incident

Imagine a support agent called Northstar that can issue refunds. Its normal run takes about 7 seconds. At 09:14, one run took 11.8 seconds and issued an unsafe refund.

The trace timeline looked like this:

Span	Duration	Finding
model.plan	2.1 s	1,180 input tokens, 220 output tokens
retrieve.policy	4.6 s	vector search took 0.2 s; reranking took 4.2 s
tool.lookup_order	0.3 s	read-only call succeeded
approval.wait	4.4 s	approval queue was backed up
tool.issue_refund	0.4 s	side effect completed

The run was slow because retrieval and approval consumed 9 seconds of the 11.8-second wall-clock duration. The model was not the bottleneck. This distinction matters when the proposed fix is “buy a faster model,” which would improve almost nothing.

The safety failure was more subtle. The refund policy required approval for amounts above 1,000 US dollars. The model emitted an integer with a dollars label. The approval service interpreted that integer as cents, while the refund tool interpreted it as dollars. The approval therefore saw 80 dollars, but the tool issued 8,000 dollars.

The broad trace did not contain the customer’s name or the raw support conversation. It did contain a safe structured event like this:

{
  "span_name": "tool.issue_refund",
  "tool_version": "refund-2026-08-17",
  "argument_schema": "hmac:7f31...",
  "requested_unit": "dollars",
  "tool_unit": "dollars",
  "amount_bucket": "5000-10000",
  "approval_amount_bucket": "50-100",
  "policy_version": "refund-v4",
  "approval_decision": "approved",
  "action_hash_match": false,
  "validation_result": "failed",
  "outcome": "completed",
  "duration_ms": 400
}

The exact amount could live in a separate encrypted audit record with stricter access and a short retention period. The trace still exposes the important fact: approval and execution referred to different canonical actions. A validation failure before the side effect should have blocked the call. The metric unsafe_action_completed should increment separately from unsafe_action_blocked; collapsing both into policy_violation hides the severity.

Metrics without a privacy leak

Use histograms for durations and counters for outcomes. Track p50, p95, and p99 for the whole run and for each span type. Break them down by bounded dimensions such as model deployment, tool name, retrieval backend, workflow version, and region.

Avoid labels containing user IDs, raw URLs, document IDs, prompts, query text, or arbitrary error strings. Those create high-cardinality metrics, which are expensive and difficult to query, and they can become a quiet data-exfiltration channel.

Useful counters include model timeouts, retrieval-empty results, tool validation failures, approval expirations, state-loop detections, policy denials, unsafe attempts, unsafe blocks, and unsafe completions. Put trace IDs into logs or metric exemplars where the backend supports it, so an alert can lead to a representative trace.

Sampling needs special care. Ordinary successful runs can be sampled aggressively. Slow runs, errors, policy denials, validation mismatches, and suspected unsafe actions should use tail-based sampling or a durable safety event so that a rare incident is not discarded before anyone sees it. The collector still needs to receive enough metadata to make that decision, and its access controls matter just as much as the application’s.

Do not solve privacy by deleting all detail. Keep sensitive content in a separate, encrypted, audited store with narrow access, explicit retention, and incident-triggered promotion. The default trace should answer “which component, version, decision, and timing failed?” It should not answer “what did every customer type this week?”

What they’ll ask next

How would you tell whether the model or retrieval is slow?
Break model latency into queue time, first-token time, and generation time, then compare it with retrieval search and reranking spans. Look at the critical path, not just the sum: parallel child spans can overlap.

How do you catch a rare unsafe action if traces are sampled?
Emit a low-content safety event for every policy decision and tool side effect. Keep full traces for violations, mismatched action hashes, failed validations, and unusually slow runs using tail sampling or an incident-triggered retention rule.

Would you log the tool arguments to debug a bad action?
Only through an allowlisted, schema-aware representation. Log types, units, ranges, validation results, redacted buckets, and keyed fingerprints in the normal trace. Put exact arguments in a restricted audit record when the risk and retention policy justify it.

One line to say in the room

“I want one causally connected trace from model decision to side effect, metrics that separate each source of latency, and approval-bound action hashes so I can diagnose failures without turning every customer prompt into a log file.”