Batch vs real-time inference

Two workloads, two contracts

At 9:07 on Monday morning, a subscription app opens its home page for 1,200 people per second.

The recommendation model needs the user’s current plan, recent clicks, and today’s inventory. A response in 300 milliseconds is useful. A response in 30 seconds is useless.

The same company also needs a churn score for 2 million accounts. Marketing looks at those scores once each morning.

Recomputing the score every time an analyst opens a dashboard would produce the same answer thousands of times, while keeping a service running all night.

One workload needs a request-triggered prediction. The other needs a scheduled table. The model may be identical. The serving contract is not.

That distinction matters more than whether the model is a random forest, a transformer, or a 7 GB neural network. Pick the wrong contract and you either pay for an always-available service that nobody needs, or make a customer wait for work that should have happened hours ago.

Start with four clocks

“Real-time” often mixes together four constraints:

• Freshness: how old may input data be? A churn score may use yesterday’s events; a fraud score may need a transaction that arrived 20 milliseconds ago.
• Response latency: how long can the caller wait after making a request?
• Throughput: how much total work must be done? Scoring 2 million accounts once per day is high volume, but not high request rate.
• Burstiness: does work arrive smoothly, or in a 10-minute spike?

A nightly job can write predictions into a database in advance, so a dashboard responds in 30 milliseconds even though the prediction is 18 hours old.

An online API can answer in 40 milliseconds while using a feature that is six hours stale. Fast response does not imply fresh data.

Choose along three separate axes:

• Input shape: a finite population, an individual request, or a continuous event stream.
• Caller contract: does the caller wait for a synchronous response, or submit an asynchronous job and collect the result later?
• Computation timing: are features or predictions precomputed, calculated from request-time facts, or split between both?

Online usually means request-triggered serving. It does not promise that the caller waits or that the facts are fresh.

Real-time is a requirement about latency, freshness, or both. A nightly churn table, an asynchronous upload endpoint, a stream processor, and a recommendation API combining cached and live features are all valid designs.

Common combinations

Batch and streaming describe incoming work. Synchronous and asynchronous describe whether a caller waits. Precomputed and request-time describe when computation happens.

Batch inference

A batch job reads a defined finite set of entities, scores them, and writes results somewhere durable. It might run nightly, hourly, or after a warehouse partition closes. The application reads a stored prediction:

customer_id | churn_probability | model_version | scored_at
             1842                 0.73              v17          2026-08-28 04:32

Batch suits:

• churn scores
• lead ranking
• risk reports
• inventory forecasts
• recommendation candidates

It uses compute when the job runs and processes records efficiently in chunks. It cannot use a decisive input that appears during the request unless another system has already incorporated it.

Online synchronous inference

An online service receives a request, obtains features, runs a warm model, and returns a response while the caller waits. It may read cached features, calculate request-time features, or combine both.

This fits:

• transaction fraud
• search ranking
• dynamic pricing
• decisions that block the next product step

Capacity must be available during quiet periods, dependencies share a deadline, and tail latency matters. p99 means the 99th percentile; one request in 100 is slower than that value.

Async inference

Async inference accepts work now and finishes it later. An HTTP caller may receive 202 Accepted and a job identifier, then poll or receive a callback. A queue separates the user-facing service from workers and absorbs bursts.

This fits:

• document extraction
• image analysis
• large uploads
• 30-second summarization

Async is a caller contract, not an input shape: a batch upload can be batch plus async, and a stream processor is usually async relative to event producers.

Streaming inference

Streaming scores a continuous flow of events. A consumer reads a durable log, maintains state by key, and emits predictions as events arrive. This suits machine-vibration windows or continuously updated payment risk.

Streaming must handle:

• out-of-order and late events
• event time
• watermarks
• replay
• offsets
• checkpoints
• state recovery

“Exactly once” processing does not make an external email, database update, or payment action happen exactly once; sinks must be idempotent too. For a once-a-day score, this stateful system is unnecessary complexity.

The arithmetic behind the choice

Worked capacity example

Suppose the subscription company has 2 million customers. Its measured batch benchmark is 5,000 rows per second. Extraction takes 20 minutes, validation 5 minutes, and publishing 1 minute.

Assume these stages are serialized and the inference rate holds for the whole population.

The home page sees a peak of 1,200 requests per second, a sustained average of 900, and a measured mean response time of 40 milliseconds, including queueing. The product contract is p99 under 200 milliseconds.

customers = 2_000_000
batch_rate = 5_000  # measured rows per second

extract_minutes = 20
validation_minutes = 5
publish_minutes = 1

peak_rps = 1_200
sustained_avg_rps = 900
mean_response_ms = 40  # mean time in the system, including queueing
p99_target_ms = 200

inference_minutes = customers / batch_rate / 60
pipeline_minutes = (
    extract_minutes
    + inference_minutes
    + validation_minutes
    + publish_minutes
)
expected_concurrency = sustained_avg_rps * (mean_response_ms / 1000)
online_calls_per_day = peak_rps * 24 * 60 * 60

print(f"Batch inference: {inference_minutes:.1f} minutes")
print(f"End-to-end batch path: {pipeline_minutes:.1f} minutes")
print(f"Expected concurrency at sustained average: {expected_concurrency:.1f}")
print(f"Calls if peak rate held all day: {online_calls_per_day:,}")
print(f"Online p99 target: under {p99_target_ms} ms")

It prints:

Batch inference: 6.7 minutes
End-to-end batch path: 32.7 minutes
Expected concurrency at sustained average: 36.0
Calls if peak rate held all day: 103,680,000
Online p99 target: under 200 ms

Read the timing

The batch path takes 32 minutes 40 seconds. Starting at 04:00, publishing finishes around 04:32:40, leaving 87 minutes 20 seconds before a 06:00 deadline.

That is schedule slack, not freshness.

If the immutable input snapshot has an as_of timestamp of 03:00, a dashboard read at 05:00 sees data two hours old; at 09:00, it sees data six hours old. Freshness is measured from read time and input cutoff, not pipeline duration.

Little’s law says average items in a stable system equal arrival rate times average time in the system:

900 requests/second × 0.040 seconds = 36 expected concurrent requests

That is an average, not peak capacity or a guarantee.

At 1,200 requests per second and an actual 40-millisecond duration, 48 requests would be a rough in-flight count. Bursts, failures, and saturation require additional capacity and load testing.

Halving the mean response time to 20 milliseconds reduces the estimate to 18; changing only the p99 target does not change measured concurrency.

There is no universal claim that batch is “ten times cheaper.” Cost depends on utilization, hardware, model runtime, storage, network, and the cost of being wrong or late. These numbers reveal capacity; they do not replace an SLO or a pricing estimate.

Production patterns

Make batch runs publishable

A reliable batch path:

1. Captures an input snapshot or clear cutoff and records feature as_of.
2. Computes into a run-specific location containing run_id, model_version, and scored_at.
3. Validates row counts, null rates, schema, score ranges, and distributions.
4. Publishes one complete run by updating a pointer or swapping a table view.
5. Retains metadata needed to reproduce or explain the result.

Writing directly over the production table can leave a dashboard with 61 percent of today’s customers and 39 percent of yesterday’s.

Retries should be idempotent: rerunning the same input and model replaces or reuses its run instead of appending duplicates. See data contracts and data and model versioning.

Keep online paths short

A typical request is:

validate request → fetch or calculate features → run warm model → apply policy

Give every dependency a time budget.

Track:

• p50, p95, and p99 latency
• timeouts
• concurrency
• queue depth
• feature age
• errors
• model version

A fallback is a business decision: a last-known recommendation score may be acceptable, but failing open on payment risk may not be.

Using different feature transformations in training and serving is training-serving skew. Compare transformations and timestamps across both paths.

Make async jobs and streams recoverable

A queued job needs:

• a stable identifier
• visible status
• retry limits
• a permanent-failure location
• an idempotency key

Queue redelivery after a worker crash is normal. If jobs arrive at 500 per second and workers finish 100 per second, the queue grows by 400 per second; 202 Accepted does not change that arithmetic.

For streams, define whether timestamps mean processing time or event time. Persist offsets and state.

Monitor:

• consumer lag
• event-time delay
• watermark progress
• state-store size
• age of the last prediction

A stream can have healthy process uptime while its predictions are 45 minutes old.

The hybrid that often wins

Hybrid systems split the work deliberately:

• Batch computes expensive, shared features or candidate predictions.
• A fast store holds the latest complete result and its age.
• A thin online layer reads it and adds request-time facts.
• A live model, if needed, reranks or adjusts instead of rebuilding everything.

For example, batch can calculate a user embedding while the request adds current inventory and page context.

This gives low response latency without repeating shared computation, but it does not provide unlimited freshness. Store feature_as_of and prediction_age; the caller can enforce a freshness budget or choose a stale-data fallback.

A feature store helps when many models and consumers need consistent offline and online features. For one nightly table and one dashboard, a versioned warehouse table is often simpler. See feature stores for the boundary.

Match the contract to the pattern

A billion records once a week may be batch. Ten requests per second during a card authorization may require synchronous serving. Volume alone does not choose the architecture.

Failure modes: symptom first, then fix

Monitor prediction age and completeness, not only process health.

A successful response can still violate the product contract if its features are obsolete.

What to remember

Freshness is the age of the facts. Latency is the time the caller waits. Throughput is the amount of work. Burstiness is when that work arrives.

• Batch avoids repeated computation.
• Online avoids waiting for a scheduled result.
• Async absorbs work the caller does not need immediately.
• Streaming keeps a signal current but adds state and replay complexity.
• Hybrid precomputes shared work, records its age, and adds live context in a thin online layer.