ML observability and SLOs

ML observability and SLOs

At 02:17, a card-fraud model starts seeing transactions with no device_id.

The service still returns HTTP 200. CPU is at 41 percent. Memory is normal. Median latency is 38 milliseconds. The dashboard is green.

But the missing value is replaced with a harmless-looking default. The model becomes less certain, then starts allowing transactions it would previously have blocked. Chargebacks arrive ten days later, after the on-call engineer has stopped looking at the graphs.

This is the awkward part of machine learning in production: a request can be fast and technically successful while the answer is getting worse.

Observability is the ability to infer what a running system is doing from the signals it emits. An ML service needs signals for infrastructure, serving, input data, and model behavior.

It also needs a promise: which failures matter enough to wake someone, and how much unreliability can the team spend while shipping a new model? That promise is an SLO, or service-level objective: a measurable target for a service over a stated period.

Without that promise, monitoring becomes a museum of interesting charts. Everyone looks. Nobody knows what requires action.

One model, four layers of telemetry

Our running example is a fraud model that returns a score from 0 to 1. Transactions with a score of 0.82 or higher are blocked for review.

Telemetry means the measurements and records emitted by a system. Its four useful layers catch different failures.

1. Infrastructure: can the machinery keep up?

Track:

• CPU and memory, plus GPU usage when applicable
• container restarts and out-of-memory kills
• queue depth and replica count
• network and disk errors

If a container runs out of memory, requests fail before inference. If a queue grows, waiting time grows. If the feature store is unreachable, the service may return defaults or errors.

Infrastructure health is necessary but not sufficient: a healthy machine can serve unhealthy predictions. Tag metrics with service, environment, region, and model version. Avoid user and transaction IDs in metric labels; excessive cardinality can create its own incident.

2. Service: can users get a valid answer in time?

Track latency, error rate, throughput, timeouts, and invalid or incomplete responses. Report latency percentiles such as p50, p95, and p99; averages hide the slow tail. Count application failures, not only transport failures: HTTP 200 with no score or an out-of-range score is not a successful prediction.

Throughput is also a signal. A sudden drop may indicate an outage; a sudden jump may indicate retries, a bot, or a broken upstream job.

3. Data: are inputs what the model expects?

Check:

• required fields, names, and types
• null and missing-value rates
• valid ranges and categorical values
• volume, freshness, and important feature distributions

A schema is the agreed structure of an input record. If account_age_days changes from days to seconds, the service may accept a number while feeding nonsense into the model. If country becomes mostly null, the model may rely on a fallback path. This is training-serving skew: prediction-time data differs from what the model learned from.

Validate before inference when possible. Reject malformed records or use a controlled, separately counted fallback. Silent defaults hide the failure in the model layer.

4. Model: is decision behavior changing?

Track prediction counts, score and confidence distributions, fallback rates, output violations, important slices, and eventually precision, recall, loss, calibration, or business outcomes.

A prediction distribution shows how outputs change. If the fraud model usually blocks 6 percent of transactions and suddenly blocks 18 percent, something changed. It might be a fraud wave, a bad feature, or a new model; the distribution tells you where to investigate, not which explanation is true.

Confidence is not accuracy. Calibration means confidence values line up with observed correctness, but a model can still be confidently wrong. Quality telemetry requires ground truth: in this case, a later chargeback or investigation outcome.

Keep quality grouped by prediction date and model version, not only by label-arrival date.

The diagnostic sequence is:

• Infrastructure: is the runtime healthy?
• Service: do callers receive valid answers on time?
• Data: do inputs match the contract?
• Model: is behavior and eventual quality acceptable?

Do not collapse these into one “model health” number.

The delayed-label problem

The model answers now. The truth often arrives later. A cardholder may dispute a transaction after 14 days; a recommendation purchase may happen three days after a click; lending outcomes can take months.

Use two clocks:

1. Prediction time: when the model made the decision.
2. Label time: when the outcome became trustworthy.

For a 14-day fraud label, evaluate cohorts at least 14 days old. Keep model version and important slices attached to each cohort. Do not calculate today’s precision from labels that arrived today; that mostly measures older transactions and can misclassify recent unlabeled traffic.

Before labels mature, use proxy signals such as:

• missing device_id rising from 0.2 percent to 4 percent
• blocked share moving from 6 percent to 18 percent
• median confidence falling from 0.91 to 0.62
• score shifts by country or a rise in fallback predictions
• increased review volume or customer complaints

These find change but do not prove quality has fallen. A genuine fraud wave can increase the blocked share.

This is where drift detection belongs: as a leading indicator and investigation aid, not proof that the model is wrong.

SLIs, SLOs, and error budgets

An SLI, or service-level indicator, is the measurement:

successful eligible prediction requests divided by all eligible prediction requests

An SLO applies a target over a time window:

99.5 percent of eligible prediction requests return a valid response over 30 days

An error budget is the permitted failure. It turns “be reliable” into risk the team can spend on releases and experiments.

Define the denominator first. “All requests” might include health checks, load tests, or malformed internal calls. Exclusions can be reasonable, but they must be explicit and monitored.

For a request-based SLO, suppose the service handles 1,200,000 requests and 8,000 fail:

(1,200,000 - 8,000) / 1,200,000 = 99.333 percent

The service misses its 99.5 percent SLO. Its allowed failure fraction is 0.5 percent, or 6,000 requests, so it exceeded its budget by 2,000 failures.

Do not confuse that with a time-based budget. A 99.5 percent availability SLO over a 30-day month permits:

• 30 × 24 × 60 = 43,200 minutes
• 0.005 × 43,200 = 216 unavailable minutes

A two-hour outage uses 120 minutes, or 55.6 percent of that time budget. A request-based budget depends on traffic: at 100 requests per minute, the same outage affects 12,000 requests.

Use separate SLOs for separate promises:

• Availability: 99.5 percent of eligible requests return a valid decision.
• Latency: 99 percent complete within 300 milliseconds.
• Quality: at threshold 0.82, mature labeled transactions meet agreed precision, recall, and action-rate or expected-cost targets.

Precision is the share of blocked transactions that are fraudulent. Recall is the share of fraudulent transactions that are blocked. The action rate is how much traffic enters the costly or disruptive path.

A quality SLO needs a label definition, maturity window, cohort rule, fixed operating point, minimum mature sample, and slice policy. Precision alone is unsafe: raising the threshold can block fewer, easier cases and make precision look better while recall falls. Pair precision with recall or false-negative rate and coverage or cost. Report confidence intervals when samples are small.

Do not average availability, latency, and quality into one score. A service can be fast and available while making poor decisions.

If the service has spent 95 percent of its budget on outages, pause a risky model rollout and fix reliability. If it has ample budget, a canary or retraining experiment may be reasonable. The budget informs the decision; it does not make it automatically.

Alerting without alert fatigue

An alert should answer:

1. What is happening?
2. Who needs to act?
3. How quickly?

Page on customer symptoms:

• rapid error-budget burn
• sustained failures or invalid decisions
• p99 latency violating its SLO
• a queue growing fast enough to threaten availability

Create a ticket or notification for leading indicators:

• schema or null-rate changes
• score or confidence distribution shifts
• unusual volume
• slow drift in a region or segment
• mature-label quality decline without an immediate outage

Dashboard-only metrics are useful context without a clear action. CPU at 65 percent with normal queue and latency is a trend, not an incident.

For delayed quality, page only when the breach is severe and credible, the mature sample and label coverage are sufficient, labels are fresh for the cohort, and there is immediate mitigation such as rollback or disabling a feature. Otherwise assign an investigation with evidence and a review time.

Signal	Page now	Ticket or notify	Why
High-rate failures or invalid decisions	Yes	No	Customers are already receiving failures
p99 latency breaching its SLO	Yes	No	Users experience waiting or timeouts
Null or score-distribution shift with stable service	No	Yes	It is a leading indicator
Mature-label quality below SLO	Only with a severe, credible breach and mitigation	Yes otherwise	Quality needs mature evidence and a response

Group related alerts. One broken feature pipeline may cause null-rate, confidence, distribution, and quality alerts. The incident response process should name the owner, rollback path, and evidence to collect.

Correlation IDs: joining today to two weeks from now

A correlation ID is a unique identifier carried across related events. Give each prediction one and store it with the prediction timestamp, model and feature versions, score, decision, validation result, region, and latency.

When the outcome arrives, store the same ID with the label, label timestamp, and source. The quality job can then join the prediction to its outcome without guessing. A distributed trace ID follows a request across services; a prediction ID follows the model decision into a later business event. Use both when needed.

from datetime import datetime, timezone
from uuid import uuid4
import json

def now():
    return datetime.now(timezone.utc).isoformat()

prediction_id = uuid4().hex
print(json.dumps({
    "event": "prediction",
    "correlation_id": prediction_id,
    "prediction_time": now(),
    "model_version": "fraud-model-2026-08-29",
    "score": 0.95,
    "decision": "block",
    "valid_response": True,
}))

# Later, when a trustworthy outcome arrives:
print(json.dumps({
    "event": "outcome",
    "correlation_id": prediction_id,
    "label_time": now(),
    "fraudulent": True,
    "label_source": "chargeback",
}))

In production, use durable storage with retention longer than the label delay. Make outcome ingestion idempotent so duplicate events do not count twice. Do not log raw card details or sensitive features merely because they are convenient; record carefully selected aggregates and feature versions instead.

Observability has an honest limitation: it can show that behavior changed but cannot always explain why. A confidence shift may come from population change, a feature bug, a new model, or an upstream launch. Diagnosis still needs logs, ownership, controlled rollback, and sometimes individual records. Telemetry also costs money and creates privacy obligations, so instrument fields that support a decision.

What to remember

• Infrastructure, service, data, and model telemetry catch different failures.
• Ground-truth quality is delayed: use proxies for warning and mature cohorts for evaluation.
• An SLI is the measurement, an SLO is the target, and an error budget is permitted failure.
• Page on customer symptoms and rapid SLO burn; investigate drift and null-rate changes without waking people unnecessarily.
• A prediction-specific correlation ID connects today’s score to a later outcome.

Quick check

0/3

Q1

An infrastructure outage that should always page immediatelyA data-layer leading indicator that should trigger investigationProof that model precision has fallenA successful request because the HTTP response was valid

Check answer

Q2

21.6 minutes216 minutes2,160 minutes4,320 minutes

Check answer

Q3

Page the whole on-call team for confirmed quality failureIgnore it because no ground-truth labels have arrivedOpen a leading-indicator investigation, check the affected data path, and wait for mature cohorts before judging qualityRetrain immediately using the unlabeled clicks as negative examples

Check answer