Batch can be 100x cheaper for low-utilization, predictable workloads

At 3 a.m., your churn model has no traffic. Nobody is waiting for a customer score. The server is still holding model weights in memory, passing health checks, collecting logs, and waiting for the morning spike.

At 8:00, a lifecycle job needs scores for 50,000 customers. The model will process the same kind of data it processed yesterday: recent usage, payment status, support history, and account age. Nothing about the decision requires a human to wait for an HTTP response.

Yet teams routinely build a live prediction API for this job.

That is not maturity. It is usually a capacity-planning mistake.

Batch inference means scoring a known set of records together on a schedule. Real-time inference, also called request-time inference, means calculating a prediction as each request arrives, inside the request’s latency budget.

My default is simple: use batch unless a named decision needs current or request-specific information that cannot be precomputed within the freshness and response contract.

The important distinction is between two clocks. One clock measures how fresh the data must be. The other measures how quickly the caller needs an answer. Most systems pay heavily to make the second clock fast when the first clock is allowed to be slow.

Start with the decision clock

Consider Northstar, a subscription business with 50,000 active customers. Its churn model produces a probability between zero and one. The marketing team uses that probability at 8:00 each morning to choose whom to email.

Northstar does not need a live endpoint. A score computed at 1:00 a.m. is perfectly useful at 8:00. A score from the previous night is not ideal forever, but it has a clear and acceptable lifetime.

That acceptable age is the freshness budget: the maximum age of the information a decision may use.

• If the budget is 24 hours, a nightly job may be enough.
• If it is 5 minutes, you may need streaming updates or frequent micro-batches.
• If it is 200 milliseconds, you usually need continuously updated features or scores and a low-latency lookup path.

That does not, by itself, require per-request inference. A streaming pipeline, which processes updates as they arrive, or frequent materialization, which writes prepared scores ahead of requests, can meet the target. Request-time inference is needed when current or request-specific state cannot be precomputed within the contract.

The caller’s requirement is a separate response budget: the maximum time the caller will wait. A web page might need an answer in 100 milliseconds. An overnight email job may happily wait 30 minutes. A fraud decision at checkout may have only a few hundred milliseconds.

A fast response does not imply a fresh prediction. Northstar’s customer page might need to display the score instantly, while the score itself may be eight hours old. That is a fast lookup problem, not necessarily a real-time inference problem.

This is the first design question to put in the architecture document:

How old may this prediction be, and when is the decision made?

Do not begin with, “Should we deploy the model behind an API?” That question has already smuggled in the answer.

Where a 100x worker-capacity gap can come from

The title’s 100x is not a law of physics. Batch is not magically cheap, and an API is not automatically expensive. The gap appears because batch pays for compute while it is working; a low-latency service pays to remain ready for the worst moment.

The calculation below is a worker-capacity illustration, not a billed-dollar estimate. It shows how a large utilization gap can appear under one specific set of assumptions.

Use a deliberately plain example. Suppose measurement on your chosen worker shows that one prediction consumes 30 milliseconds of worker CPU time. Scoring 50,000 customers therefore consumes:

50,000 x 0.03 seconds = 1,500 seconds = 25 minutes

A batch job can run one worker for those 25 minutes, then release it.

Now suppose the production API requires two worker replicas for availability. Each replica stays available for 1,440 minutes per day. That is:

2 x 1,440 = 2,880 worker-minutes per day

Compared with 25 minutes of batch work, the always-on service uses:

2,880 / 25 = 115.2

That is the rough shape of the 100x claim. The model computation is the same. The difference is idle capacity, replica overhead, and the need to be ready before traffic arrives.

This is a worker-capacity illustration under deliberately narrow assumptions. It is not a claim that your cloud bill will be 115.2 times higher, or that batch is universally 100x cheaper.

It leaves out or holds aside:

• scheduler and orchestration charges
• warehouse reads and writes
• durable storage
• Redis
• startup and teardown
• retries
• batch parallelism
• scale-to-zero behavior

A billed-dollar model must add those costs, plus the price of the actual instance or managed service and any network transfer.

Batch parallelism can reduce wall-clock time but does not automatically reduce total worker-minutes. An endpoint that scales to zero can remove idle replicas, but it can also add cold-start latency and still incur request or platform charges.

The headline is a warning to measure utilization, not a number to paste into a business case.

Size for the peak, not the average

Daily averages also hide the peak. Northstar may average only 0.58 predictions per second across a day.

If 25,000 customers open the product during a ten-minute campaign, the service must absorb about 42 predictions per second during that window. A real-time deployment is sized for that burst, not for the comforting daily average.

Batch can queue the same work. It can process the records in a larger batch, use a machine when it is available, and retry a failed partition without keeping a public endpoint alive.

The trade is waiting for the result. If the business already waits until morning, that trade costs nothing.

Measure before choosing

The ratio changes when the assumptions change. A tiny model behind an endpoint that scales to zero may be cheaper than maintaining a scheduler, warehouse job, and cache. A high-volume API may keep its hardware busy enough that idle time is small. And if every request needs a new piece of information, batch cannot replace the computation at all.

So treat 100x as a warning to measure utilization, not as a promise to paste into a business case.

Make the lookup real-time, not the model

There is a useful middle path.

Precompute the result

Northstar can run the model at 1:00 a.m. and write the complete output to durable storage, the source of truth that survives cache eviction and service restarts.

That might be an indexed database table or an object-store artifact. It can then populate Redis, an in-memory key-value store, with keys under a run-specific namespace. The customer page reads that result when it loads. The page gets a quick response. The model does not run during the request.

This is the precompute-to-cache pattern:

1. A scheduled job reads a defined data snapshot.
2. It scores the records and writes a complete, durable result with a run ID.
3. It populates Redis under a run-specific namespace and validates the complete run.
4. It atomically updates the active-run pointer. The serving layer resolves that pointer, looks up the result by customer ID, and returns the score with its age.

The API’s work is now a lookup, pointer resolution, serialization, and response. In a nearby deployment, that can take a few milliseconds without loading a model or waiting for feature computation.

The latency is real. The inference is not.

Keep a durable source of truth

Redis is the serving cache, not the durable record. Configure Redis persistence and replication to match the recovery target.

Treat eviction, restart, or failover as a cache miss, then rebuild the active namespace from the durable batch output. If a rebuild fails, keep the previous active run while it is still within the freshness contract; otherwise follow an explicit fallback policy.

Do not let a successful Redis connection masquerade as a complete prediction store.

This pattern resembles the online side of a feature store. A feature store usually keeps historical feature data in an offline store and makes selected, recent features available through a low-latency online store.

A precomputed prediction in Redis is not the same thing as an online feature, but the split is similar: expensive preparation happens away from the request path, and the request path retrieves a prepared value. The feature store pattern explains the online and offline boundary in more depth.

Do not add Redis automatically. If Northstar has 50,000 rows and the page can tolerate 50 milliseconds, a database table with a suitable index may be easier to operate. Redis earns its place when lookup latency, traffic volume, or isolation from the primary database justifies another moving part.

Fast does not mean fresh

The most dangerous misunderstanding in this design is calling a cache hit “real-time inference.”

Reproduce the same score

To expect batch and online predictions to match, use the same model artifact and preprocessing, the same feature values and availability cutoff, and deterministic execution. Different pipelines may still occasionally produce the same score by coincidence.

Preprocessing is the set of transformations applied before scoring. An availability cutoff is the latest time whose data the pipeline was allowed to use. Neither is necessarily the same as the time an event was created.

Point-in-time correctness means that a prediction may use information available at its prediction timestamp, but not information that arrived later.

Suppose Northstar computes an 8:00 a.m. online score during an experiment. The nightly batch score, published at 1:00 a.m., uses a snapshot with a data-availability cutoff of midnight. That snapshot includes a support ticket that became available to the pipeline at 11:50 p.m.

The online feature pipeline is lagging. When the 8:00 a.m. request is served, its latest published cutoff is 11:00 p.m., so the online score omits the ticket.

An up-to-date online pipeline would normally have access to it. The difference here is feature-pipeline lag, and therefore a training-serving skew problem, not an ordinary point-in-time difference between batch and online scoring.

Store enough metadata to explain what happened. A prediction record might look like this:

{
  "customer_id": "c_1842",
  "score": 0.73,
  "model_version": "churn-2026-08-27",
  "features_as_of": "2026-08-27T23:00:00Z",
  "computed_at": "2026-08-28T01:12:04Z",
  "expires_at": "2026-08-28T23:00:00Z"
}

With a 24-hour feature-age budget, the features_as_of value expires at 2026-08-28T23:00:00Z, exactly 24 hours after the feature cutoff.

It does not expire 24 hours after computed_at.

Publish within the freshness contract

That clock matters for publication. If the next similarly timed run does not publish until 2026-08-29T01:12:04Z, the old artifact has already expired, leaving a 2-hour-and-12-minute gap.

Northstar must publish the next artifact before expiration if it promises continuous validity, or define a deliberate overlap or fallback policy. It might allow the previous score during that gap but mark it stale, return “unavailable,” or queue a refresh.

If the policy is instead based on score age or the next publication window, state that explicitly and derive expires_at from that contract.

A cache TTL is useful for cleaning up memory. It is not a freshness policy. If the nightly job fails and the cache keeps yesterday’s value, the key may still exist and the API may still return a successful response.

The system looks healthy while serving stale decisions.

The first symptom of a broken batch pipeline is often not an error. It is a stream of perfectly normal 200 responses whose computed_at timestamp has stopped advancing.

A second common symptom is a sudden cache-miss spike after someone changes the key format or model version. The API then falls back to live inference for every miss, turning a cheap lookup path into a traffic amplifier.

Switch runs atomically

Publish results atomically. Keep the batch output in durable storage. Write Redis keys under a run-specific namespace, such as predictions:2026-08-28T01:12:04Z:c_1842.

Validate the complete run’s row count, missing IDs, and score distribution. Then atomically update one active-run pointer, such as predictions:active_run.

The API must resolve that pointer on every read before constructing the customer key. A pointer switch is what makes the population change atomic; writing thousands of versioned keys alone does not.

Without the pointer, readers can observe a mixture of old and new runs. If Redis is being rebuilt, keep the old pointer until the new namespace passes validation.

On a miss or an expired value, use the documented fallback rather than silently switching every request to live inference.

For disagreements between batch and online scores, compare feature values and timestamps before changing the model. Training-serving skew is often a data assembly problem wearing a model-shaped costume.

When real-time is the right answer

Request-time inference is necessary when the request carries current or request-specific information that cannot be precomputed within the freshness and response contract.

A payment fraud model needs the current transaction amount, merchant, device fingerprint, and recent attempts.

A search ranker needs the query the person just typed and the inventory available now.

A dynamic pricing model may need current supply and demand.

Waiting until tomorrow is not a graceful degradation for those decisions.

There are also decisions where a short delay changes the outcome. If an account takeover can happen in 30 seconds, a daily risk score is not a control. If a recommendation only changes after the next morning’s catalog refresh, a live endpoint is mostly theatre.

Streaming can sit between the two. A pipeline may update scores every minute or every five minutes, then serve them from a store. That is fresher than nightly batch and cheaper than recalculating for every request.

Name the actual freshness budget rather than calling every frequent job “real time.”

The strongest argument for the API-first approach is not technical. It is operational simplicity.

For a low-volume model, one endpoint may be easier for a small team than a scheduler, a backfill process, a data-cutoff check, a result table, cache publication, and freshness monitoring. The endpoint can also avoid stale-cache behavior because it computes from current inputs.

That is a legitimate reason to choose an endpoint even when a precomputed score could meet the freshness contract. It is a separate engineering reason, not evidence that every request needs fresh inference.

That argument is fair. Batch is not free. It creates:

• scheduling problems
• data-cutoff problems
• retry problems
• publication problems

It may require a second serving artifact. If the score is cheap and the decision genuinely needs current data, use the endpoint.

But do not confuse fewer visible components with less operational risk. A live model service adds:

• availability targets
• autoscaling
• model loading
• concurrency limits
• p99 latency monitoring
• deployment rollbacks
• peak-capacity planning

The complexity exists either way. Batch makes the waiting explicit; real-time hides it inside an always-on service.

What to do on Monday morning

Start with Northstar’s existing model, not a greenfield platform.

1. Write the decision contract. Record who consumes the score, when they consume it, the maximum acceptable age, and the response budget. “Fresh” is not a requirement. “No older than 24 hours at the 8:00 a.m. email send” is.

2. Measure the current workload. Count predictions per day, peak predictions per minute, model CPU time, memory use, and the cost of keeping the service available. Use the measured worker time in the calculation above. If the model spends 25 minutes working and the service reserves 2,880 worker-minutes, you have a concrete argument rather than a slogan. Track it with ML cost and FinOps.

3. Build one versioned batch artifact. Include the model version, data cutoff, row count, missing-record count, and score distribution. Fail the job when the input snapshot is unexpectedly small or late. A green scheduler status is not proof that the data was current.

4. Publish results safely. Write the new durable output and Redis keys under a new run, validate the complete run, then atomically make that run active through the pointer.

   Store computed_at and expires_at with every result. Use a cache only if the latency or traffic requirement warrants it. Keep the durable artifact so eviction, restart, failover, or a failed rebuild can be recovered without rerunning the model from scratch.

5. Make the API boring. It should resolve the active run, look up the customer, check that the result is not expired, and return the score with its metadata. On a miss or an expired value, choose an explicit policy: return “unavailable,” queue a refresh, or use a tightly bounded live fallback for a critical path. Do not silently turn every cache miss into inference.

6. Alert on age, not just errors. Monitor batch completion, source-data lag, maximum prediction age, cache-hit rate, missing keys, and the fraction of requests using a fallback. Add a test that advances the clock past expires_at and verifies that stale values are rejected.

The result is usually the best of both worlds: batch economics and a fast request path. Real-time inference remains available for the decisions that earn it.

Your model server does not become more intelligent by staying awake all night. It just becomes more expensive.

For the broader choice among batch, streaming, caching, and live inference, see batch versus real-time inference.

Check yourself