datarekha
Infrastructure June 10, 2026

Most predictions don't need real-time (and batch is 100x cheaper)

The default mental model of 'serving a model' is a live API answering in milliseconds. For most use cases that's the expensive wrong choice — and the hybrid precompute-to-Redis pattern gives you batch economics with real-time latency.

6 min read · by datarekha · mlopsinferenceservingbatchreal-time

When people picture “serving a model,” they imagine a live API answering each request in milliseconds. It’s a vivid image — and for most production ML, it’s the expensive wrong choice. The serving pattern you pick is driven by two questions: how fresh must the prediction be, and how many do you make? Get it wrong and you either miss your latency target or pay 100x more than you needed to.

Batch is the default for a reason

A large majority of predictions simply don’t need to reflect this instant. Churn scores, lead rankings, recommendation precomputes, risk models — all are fine computed on a schedule. One analysis argues roughly 90% of ML predictions don’t need real-time inference and can be served from batch, and the cost gap is dramatic: for scoring 50,000 customers daily, a batch approach ($0.75/month) versus real-time ($146/month) yields about 99.5% cost savings with identical predictions, trading instant freshness for daily. Batch is a scheduled job; real-time is an always-on, autoscaled, monitored service with strict latency SLAs. The complexity difference alone justifies asking “does this actually need to be fresh per request?” before defaulting to an API.

The hybrid that gets both

You don’t always have to choose. The dominant cost-saving pattern in 2026 is to precompute and cache: nightly-batch-score all customers, store the results in Redis, and let a real-time API return the precomputed score in under 5ms as a cache hit with no inference at request time. This is exactly what a feature store does for the online path — feature stores split serving into an online store (low-latency features for real-time inference) and an offline store, with Redis commonly chosen as the online store for ultra-low latency at high throughput. Computing features offline and writing them to Redis with a TTL lets the serving layer retrieve fresh precomputed features with sub-millisecond latency, far faster than computing them at inference time, and the cloud providers package it directly — AWS documents building an ultra-low-latency online feature store on ElastiCache for Redis for real-time inferencing.

The freeing realization: the latency you’re paying a fortune for is, most of the time, latency nobody asked for.

Skip to content