Sleep-time compute: the work an agent should do before anyone asks

There are two places to spend compute on a language-model system, and the industry spends most of its attention on the noisier one.

Test-time compute is the visible axis: sample more candidates, revise more times, search deeper. It works, and it is also the most expensive possible moment to do work, because someone is sitting there watching a spinner.

Sleep-time compute is everything you do while nobody is waiting. It is less discussed because there is nothing to demo, and it is usually the better trade — the same accuracy, paid for at a moment when latency costs nothing and capacity is cheap.

What actually belongs offline

Not everything can move. The test is whether the work depends on something only the request knows.

1. Precomputing the questions you can predict

Most corpora have a long head of questions that are entirely predictable from the content itself. If you have a document about refund windows, someone is going to ask how long the refund window is.

So generate those questions offline, answer them carefully with a large budget — full retrieval, several samples, a verification pass — and store the result. At request time you are doing a lookup and a light rewrite rather than a full pipeline.

The gain is not only latency. The offline answer can be better than anything you could afford online, because you were not paying for it in user-visible seconds. You can spend twenty times the compute on an answer that will be served ten thousand times.

The discipline this requires is a coverage measure: what fraction of live queries hit a precomputed answer, and how does that fraction move? Without it you will keep generating questions nobody asks.

2. Consolidating what happened into what it means

An agent that stores every episode accumulates a pile that gets harder to retrieve from as it grows. The fix is a scheduled pass that clusters near-duplicate episodes, promotes well-supported patterns into durable rules, merges rules that contradict each other, and evicts what is no longer earning retrieval.

This is sleep-time compute in the most literal sense — the offline reorganisation of experience into something smaller and more useful, which is exactly what makes the next request cheap.

3. Rebuilding derived artefacts

Embeddings, indexes, extracted entities, per-document summaries, denormalised joins. All of it is derived from a corpus that changes far more slowly than it is queried.

The temptation is to compute these lazily on first request, which trades a rare very-slow request for implementation simplicity. That trade is usually bad in a user-facing product, because the slow requests land unpredictably and always seem to land on someone important.

The economics are genuinely different

It is worth being explicit about why offline work is cheaper than the same work online, because it is not only about the user’s patience:

	Request path	Offline
Latency budget	Hundreds of ms	Hours
Batching	One request at a time	Batch to whatever the hardware likes
Capacity	Must be provisioned for peak	Can use the trough, or spot capacity
Failure	User sees an error	Retry, and nobody knows
Model choice	Fast enough to serve	The best one you can afford

That table is the whole argument. The same tokens bought at 3am, in a batch of ten thousand, on interruptible capacity, with free retries, cost a fraction of what they cost at 2pm one at a time — and can be spent on a stronger model.

The failure mode is staleness, not slowness

Everything precomputed is a claim that something did not change. Get that wrong and the system does not get slow, it gets confidently wrong — which is much worse, because there is no error to alert on.

Three habits keep this manageable:

Version the inputs, not the outputs. Store the hash of everything a precomputed artefact derived from. When any input hash changes, the artefact is stale by construction and can be recomputed, rather than relying on a TTL that is a guess.

Separate volatile from stable. A per-document summary is stable for months. A count of open tickets is stale in minutes. Putting both behind the same job means either recomputing summaries pointlessly or serving counts that are lies — so give them different jobs and different cadences.

Make staleness observable. Emit the age of the artefact that served each request. The moment “how old was this answer” is a number on a dashboard rather than a shrug, staleness bugs become findable.

How to choose between the two axes

The decision rule is short:

• If the work depends on this specific request, it has to be online. Spend test-time compute there, and only where you have a verifier worth spending it against.
• If the work depends only on the corpus or on accumulated experience, it belongs offline. Doing it online is paying peak prices for something you could have bought at a discount.
• If you are unsure, look at how many distinct requests would reuse the result. Reuse above one means precompute; reuse of exactly one means it was request-specific after all.

The reason this matters more for agents than for a plain chat product is reuse. An agent doing a hundred similar tasks a day is doing a hundred variations on work that was mostly decided in advance. Every piece of that you move to the quiet hours is accuracy you get for free during the loud ones.