OpenTelemetry for GenAI

At 3:07 a.m., a customer asks an agent for a refund. The agent calls the language model, searches the order database, calls the model again, and sends an email. The customer gets the wrong answer after 11 seconds.

Your log says:

request failed

That sentence is technically true and practically worthless. Which model call failed? Did the database take 10 seconds? Did the agent retry the model? Was the wrong order retrieved? Were 8,000 tokens quietly stuffed into the second prompt?

Agents make this harder than ordinary request-response software because one request becomes a small, non-deterministic program. It may branch, loop, call tools, retrieve documents, or hand work to another agent. The useful evidence is the exact sequence of work for that one request.

Observability is the practice of collecting and interpreting that evidence. OpenTelemetry, usually shortened to OTel, is the vendor-neutral open standard for producing and exporting it. The GenAI semantic conventions are the shared vocabulary for model and agent telemetry.

The important distinction is simple:

OpenTelemetry moves the evidence. GenAI semantic conventions give the evidence consistent names.

OpenTelemetry in one minute

Telemetry is data emitted by a running system about what it did. OTel deals mainly in traces, metrics, and logs.

A trace is the journey represented by correlated telemetry for one request. A span is one timed unit of work inside it: a model call, database search, or tool execution. Spans record timing, status, and metadata, and parent-child relationships form a causal tree when context propagation works.

The tree must match both temporal scope and causality. An agent-invocation span can remain open for a logical run while several operations happen. The first model call is its child. When the provider returns a tool call, that model span ends; the tool runs afterward as a sibling child of the still-open agent span. The second model call is another sibling after the tool result arrives.

Do not make a completed model span the parent of later tool work. Preserve the tool-call ID and, where useful, add a span link. A parent should remain open across both operations only when it represents the whole iteration or invocation.

Attributes are key-value fields such as:

model = claude-sonnet
input_tokens = 1200
output_tokens = 80

An event is a timestamped occurrence inside a span, such as a retry or safety rejection. A metric aggregates many requests, such as p99 duration or tokens per minute. Metrics reveal broad trends; traces explain one request.

The causal chain is:

1. Instrumentation starts a span.
2. Code or a library attaches attributes.
3. Instrumentation ends the span and records duration and status.
4. OTel exports it, usually through OTLP.
5. A collector or backend stores and displays it.

The API and SDK provide the mechanics. Exporters send telemetry; collectors can batch, sample, redact, and route it. Jaeger, Grafana, Datadog, Langfuse, and Phoenix can consume or display OTel data, though their GenAI field support varies.

OTel does not make a model deterministic. It records the nondeterminism represented in telemetry that survives instrumentation, sampling, export, retention, and ingestion.

One refund request, drawn as spans

Suppose the refund agent follows this path:

• The first model call receives 1,200 input tokens and produces 80 output tokens.
• The order.lookup tool finds the order and returns its status.
• The second model call receives 1,500 input tokens and produces 200 output tokens.
• The trace contains 2,700 input tokens and 280 output tokens in total.

Input includes more than the customer’s sentence: system prompt, history, tool definitions, retrieved text, and the current message. That is why a short question can produce a surprisingly large bill.

The tree identifies the bottleneck, shows the model-tool sequence, and exposes retries instead of hiding them inside an opaque request duration. For example, if both model calls take about two seconds but lookup takes eight, changing models will not fix the page.

A small aggregation illustrates why consistent names matter:

# A deliberately small example of aggregation after instrumentation.
spans = [
    {
        "name": "chat",
        "gen_ai.usage.input_tokens": 1200,
        "gen_ai.usage.output_tokens": 80,
    },
    {
        "name": "order.lookup",
        "gen_ai.usage.input_tokens": 0,
        "gen_ai.usage.output_tokens": 0,
    },
    {
        "name": "chat",
        "gen_ai.usage.input_tokens": 1500,
        "gen_ai.usage.output_tokens": 200,
    },
]

input_tokens = sum(span["gen_ai.usage.input_tokens"] for span in spans)
output_tokens = sum(span["gen_ai.usage.output_tokens"] for span in spans)
print(
    f"trace total: {input_tokens} input + "
    f"{output_tokens} output tokens across {len(spans)} spans"
)

trace total: 2700 input + 280 output tokens across 3 spans

The tool contributes no model tokens, but matters for latency and correctness. Use provider response usage fields whenever available; character-count estimates are only a fallback.

At illustrative rates of $3 per million input tokens and $15 per million output tokens:

2,700 / 1,000,000 × $3 = $0.0081
280 / 1,000,000 × $15 = $0.0042
total = $0.0123

Rates vary with provider, cached input, reasoning tokens, and model version. The trace supplies quantities; a pricing table supplies rates.

What the GenAI conventions standardize

A trace is portable only when fields have consistent meanings. The GenAI semantic conventions define names for common operations, including:

• gen_ai.provider.name identifies the provider.
• gen_ai.system is the deprecated legacy provider-or-system field; use gen_ai.provider.name for new telemetry.
• gen_ai.request.model records the requested model.
• gen_ai.response.model records the model that actually answered.
• gen_ai.operation.name describes chat, embedding, agent invocation, or tool execution.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens record usage.
• Tool attributes identify the tool and, when available, its call ID.
• Error and finish information describes completion or failure.

The exact fields depend on the operation, and GenAI conventions continue to evolve. Use the current OpenTelemetry registry and the version supported by your instrumentation.

Record requested and actual model values separately. An application may request model-v3 while the provider routes to a dated identifier; both fields help explain unexpected quality or cost changes. Use the agent-invocation or iteration span as the logical parent for model and tool spans. A model span normally ends when its provider response, including a tool call, arrives. Later tool work is a sibling; use its call ID or a span link for association.

A standard schema removes translation work after data exists. It does not create data, validate its truthfulness, or guarantee that every backend displays every field.

The production pattern

Start at the request boundary. Create an agent-invocation span with the current context as its parent; it is the trace root only when no inbound context exists. Put a request or trace ID in application logs when available, but never put the customer’s whole prompt in a span name or ordinary attribute.

Create spans at real execution boundaries:

1. Each model request.
2. Each tool invocation.
3. Retrieval, database, or HTTP work that can affect latency or the answer.
4. Each remote-agent invocation with propagated context.
5. Each retry attempt, under a parent for the logical operation.

A span should answer: what was attempted, which model or service handled it, how long it took, and whether it succeeded. Record model and operation at start; add response model, usage, finish reason, error type, and status at completion. Usage is often unavailable until then.

For a remote tool, the client injects OTel trace context into HTTP or messaging metadata. The tool service extracts it and creates a child span. Without propagation, the tool may succeed but appear as a separate trace: the operation worked, while observability did not.

Metrics complement spans. Export request duration, token usage, error counts, and tool latency distributions. Metrics alert across millions of requests; traces explain individual cases.

Derive cost from token counts and dimensions such as provider, model, operation, and environment. Apply a versioned pricing table outside the trace so historical costs can be recalculated.

Sampling controls storage and cost. Head sampling decides early, while tail sampling can retain errors, slow traces, or retries after seeing the whole trace. Tail sampling requires the collector to hold the relevant trace. Retain enough unsampled or specially selected data to investigate serious incidents.

Data you should not casually record

Prompts and completions can contain personal data, credentials, internal instructions, or customer documents. A trace backend is not a transcript archive.

Keep ordinary spans to low-cardinality fields: provider, model, operation, deployment, environment, status, and broad error type. Cardinality is the number of distinct values in a field. A model name is low-cardinality; a raw prompt, email address, or order ID may have millions of values, making indexes expensive and exposing data.

Capture content only for a specific debugging or evaluation need, with access controls and retention limits. Prefer omission, redaction, tokenization, or a reference to a protected store. A hash is a pseudonym, not redaction or anonymization: it can be dictionary-attacked, remain linkable, and still be sensitive. If deterministic correlation is necessary, use a keyed HMAC and treat the result as sensitive.

Tool arguments need the same care. A refund amount may help debug policy; an authorization header belongs nowhere in telemetry.

Failures practitioners see first

The trace stops before the tool

Symptom: a model span ends with a tool request, but no tool span appears.

Cause: the tool wrapper is not instrumented, or a remote tool lacks propagated context.

Fix: span the actual tool execution and instrument both sides of a remote boundary. Verify that the child span has the expected trace ID.

Every request says zero tokens

Symptom: model names and durations appear, but usage is zero or unknown.

Cause: instrumentation records the request but never maps provider response usage. Streaming responses and early exceptions often lose final usage.

Fix: record usage from the completed response. For interrupted streams, preserve reported usage and mark the span incomplete when appropriate. Do not silently replace missing data with a confident estimate.

Costs are exactly double what finance reports

Symptom: a trace query reports twice the expected tokens, often after adding middleware.

Cause: the same response is recorded on a client span and wrapper span, or a parent aggregate is summed with its child attempts.

Fix: sum authoritative leaf model-call spans, one per actual provider request. Give wrappers timing and outcome without copying usage onto them unless queries exclude descendants. Count real retries once per attempt.

The slowest operation looks fast

Symptom: the root lasts 12 seconds, but visible children total only four.

Cause: uninstrumented queue time, orchestration, serialization, network setup, or retry sleep. Sampling may also have removed children while retaining the parent.

Fix: compare root duration with child coverage, add spans around meaningful orchestration and queue boundaries, and check sampling and exporter drop counts. A trace proves only what reached the backend.

The honest trade-off

OTel adds code paths, exported bytes, collector infrastructure, and operational decisions. A one-off script may not need a distributed trace tree; a customer-facing agent that issues refunds or changes records usually does.

The standard also does not guarantee identical dashboards. Backends may preserve attributes without rendering them, ignore events, or support different convention versions. Exporters can drop fields, collectors can redact them, and SDK integrations can disagree. Validate a test trace and query the backend you operate.

The payoff is narrower uncertainty: “the model chose the wrong tool,” “the tool returned stale data,” and “the second prompt became enormous” require different fixes.

Quick check

Next

Use agent observability for tracing, alerting, and investigation. Then connect the same telemetry to cost and latency control and agent evaluation: traces are evidence both systems need.