Diffusion language models: when AI writes text all at once

At 3:00 a.m., an incident bot is asked to turn 40 lines of logs into a 500-token summary. The first sentence appears quickly. Then the rest arrives in a long, apologetic drizzle: one token every 40 milliseconds. The engineer is not waiting for insight. They are waiting for punctuation.

At 25 tokens per second, those 500 tokens take 500 / 25 = 20 seconds after generation begins. A faster GPU helps, but ordinary ancestral decoding still has a causal dependency: it accepts token 87 only after the earlier accepted context exists. Speculative or blockwise methods can propose or verify several tokens during one target-model iteration, reducing the number of serial rounds without removing the dependency among accepted tokens.

That is the problem diffusion language models attack.

The important claim is not that they make language magically simultaneous, or that autoregressive models are obsolete. It is narrower and more useful: when the answer is long, the hardware is parallel, and the quality is good enough, replacing token-by-token decoding with a small number of whole-sequence refinement passes can change the latency equation.

Most explanations stop at “all at once.” That phrase is catchy and slightly misleading. Diffusion models do not usually produce a perfect 500-token answer in one forward pass. They produce a rough, partly masked answer, inspect the whole sequence, fill or revise many positions, and repeat. The serial unit changes from “one token” to “one denoising pass.”

That distinction is the whole story.

The expensive part of an autoregressive model is the dependency chain

An autoregressive model generates the next token, where a token is a small text unit such as a word piece, from the tokens already generated. It factorizes an answer, meaning it assigns a probability to each next token conditioned on the earlier tokens, into a chain:

token 1 → token 2 → token 3 → ...

At the first step, the model sees the prompt and predicts token 1. At the second step, it sees the prompt plus token 1 and predicts token 2. The third step includes token 2, and so on.

A transformer can calculate many things in parallel, but ordinary ancestral generation still follows this chain. A causal mask, which hides future positions from each position, enforces the information rule. When generating token 2, the model is not allowed to peek at token 3 because token 3 has not been generated yet.

This is easy to miss because training looks highly parallel. During training, the model receives the entire known answer. It can process all positions in a large matrix operation, while the causal mask makes sure each position only learns from earlier text.

During inference, however, the answer is not known. Ordinary ancestral decoding has to walk through the positions one by one.

That is why “transformers are parallel” and “autoregressive decoding is serial” are both true.

The parallelism is inside each step. The steps themselves depend on one another.

For the incident bot, suppose the model has a time to first token of 500 milliseconds and then decodes at 25 tokens per second. The visible response takes about 20.5 seconds.

At 100 tokens per second, the same response takes about 5.5 seconds: 5 seconds to decode plus 0.5 seconds to first token. The first number is prompt processing and network overhead; the second is token generation. They are different costs and should be measured separately.

An attention key-value cache, usually called a KV cache, reduces some repeated work. After the model processes earlier tokens, the server stores intermediate attention data for reuse.

On the next decode step it does not rebuild every earlier key and value from scratch.

That cache is a major reason modern autoregressive serving works as well as it does. It does not, however, make token 2 independent of token 1. The server still needs the newly generated token before it can run the next step.

A cache makes each link in the chain cheaper. It does not break the chain.

Diffusion moves the serial work into refinement passes

Diffusion models borrow a broad idea from image generation: begin with a corrupted object and repeatedly make it more coherent.

Gaussian noise cannot be applied directly to categorical token IDs, which are labels rather than points in a continuous space. Some text models diffuse continuous embeddings or latents.

In a masked discrete diffusion model, a timestep-dependent forward process—a corruption process whose severity changes with the noise step—replaces tokens with [MASK]. The reverse model, trained to undo that corruption, predicts clean tokens or transitions between noise levels.

Generation follows that reverse schedule by unmasking or resampling positions. Iterative masked infilling without such a schedule is related but is not automatically diffusion.

At training time, a clean sentence might become:

The [MASK] sat [MASK]

The model learns to predict the missing pieces. It may see the word before a blank and the word after it. That bidirectional context is the important departure from ordinary causal decoding.

At generation time, a mask-based diffusion model can begin with something like:

[MASK] [MASK] [MASK] [MASK]

It evaluates the positions together. Perhaps the first pass fills the high-confidence positions:

The [MASK] sat [MASK]

A later pass uses those visible words to fill more gaps:

The cat sat down

The exact schedule differs between masked discrete models:

• Some fill only the most confident positions.
• Some resample tokens that were already filled.
• Other text-generation families use insertion-based generation or diffuse continuous embeddings and latents rather than categorical masks.

They share the broad refinement idea but are separate model families. For the masked discrete family, the useful mental model is stable: many positions are proposed in parallel, then the sequence is refined over several passes.

The phrase “all at once” is therefore shorthand, not a literal description of one perfect prediction. In a given pass, the model can score every masked position at the same time.

The prediction chosen for position 2 does not normally become visible to position 3 until a later pass. Refinement creates the dependencies gradually.

That gives diffusion models a different serial depth. With ordinary ancestral decoding, an autoregressive answer of 500 tokens needs roughly 500 serial decode steps.

A diffusion answer may use, for example, eight refinement passes. Eight is not a universal constant; quality settings and model families differ. The point is that the number of passes is often much smaller than the number of output tokens.

There is one practical complication: the model needs somewhere to put the output. A system may predict a length, reserve a mask budget, or use an architecture that grows the sequence through insertions.

If it reserves 512 positions for an answer that needs 210, it must handle the unused positions cleanly. If it reserves too few, it risks truncation. “Parallel” does not mean “length-free.”

Why the speedup can be dramatic

A simple latency comparison

Consider two deliberately simple profiles.

An autoregressive server produces 25 tokens per second after a 500-millisecond first-token delay. A 400-token answer takes about 16 seconds to decode, or 16.5 seconds including that initial delay.

Now imagine a diffusion server whose measured full-sequence pass takes 80 milliseconds and whose schedule uses eight passes. The refinement work is 8 × 80 milliseconds = 640 milliseconds. Add the same 500 milliseconds of prompt and network overhead, and the illustrative end-to-end time is about 1.14 seconds.

Those numbers are an example, not a benchmark. They show the mechanism. The first server performs 400 dependent decode steps. The second performs eight dependent refinement steps, with each step handling the sequence in parallel on the accelerator.

Advertised throughput is not a benchmark

Inception Labs advertises generation rates above 1,000 tokens per second for Mercury 2. A third-party deployment overview and speed analysis discuss that claim.

Before treating it as a benchmark, require a primary model card, API document, or reproducible benchmark that states:

• the model version
• the hardware
• the concurrency
• the prompt and output lengths
• the number of passes
• the quality target
• whether the figure is per-request completion throughput or aggregate batch throughput

Without those conditions, it is a vendor-advertised result, not an apples-to-apples comparison.

The responsible reading is “this is a vendor-advertised result whose conditions need checking,” not “every diffusion request is ten times faster.” A rate of 1,000 tokens per second means that the generation-only portion of a 400-token answer took about 0.4 seconds.

It says nothing by itself about:

• prompt processing
• queueing
• network time
• retries
• output validation
• the time before the first stable text appears

It also may be an aggregate throughput number. A server processing many requests together can keep a GPU busy in ways that a single interactive request cannot.

Always ask whether:

• the number is per request or across a batch
• the output length is fixed
• the comparison uses the same hardware and quality target

Parallel work still has a cost

There is another important trade-off. A conventional bidirectional diffusion pass usually reprocesses the whole sequence. It cannot reuse a causal KV cache in the same way as autoregressive decoding because changing one token changes the context seen by many other positions.

Diffusion avoids a long chain of tiny causal steps, but it pays for several large passes.

For a standard full-attention transformer, the attention work in one pass grows roughly with the square of the sequence length. The prompt and the generated answer both matter.

A short 30-token response may not benefit much from eight full passes. A long response may benefit enormously from fewer serial steps, provided the hardware can execute those full passes efficiently.

This is why “no KV cache” is neither a victory nor a disaster on its own. It may reduce the memory pressure of storing a growing causal cache, while increasing repeated computation.

The right question is not which cache exists. It is which design finishes the actual request with the required quality, latency, and cost.

The quality trade is about revision versus commitment

Revision versus commitment

Autoregressive generation commits to a prefix. Once the model has streamed the word “Tuesday,” it cannot reach backward and change it to “Wednesday” without starting over.

That commitment is a limitation, but it is also useful. The prefix is stable, easy to display, easy to moderate, and easy to feed into the next tool call.

Diffusion has the opposite shape. It can use evidence from both sides of a position and may revise an early choice after later positions become clearer.

For the incident summary, a first pass might place the service name and the likely failure mode in separate positions. Later passes can make the sentence between them grammatical. This is especially natural for rewriting, filling templates, and infilling missing text.

Revision is not automatically coherence. A pass that predicts every blank at once does not let those newly predicted tokens talk to one another until the next pass.

If the schedule commits too aggressively, several positions can become confidently wrong in a mutually reinforcing way. More passes can help, but they cost latency. A quality setting is therefore also a speed setting.

The same tension appears in reasoning. Autoregressive models get a sequence of intermediate states for free: each generated token becomes part of the context for the next token.

That does not guarantee good reasoning, but it gives the model a natural scratchpad that unfolds over time. Diffusion models can reason through iterative refinement, yet a small pass budget may encourage shallow guesses, while a large pass budget can erase the latency advantage.

Product behavior still needs evaluation

That does not make diffusion models unintelligent. It means generic benchmark quality is not enough to choose one.

Measure the task that matters:

• whether the incident summary names the right root cause
• whether generated code runs
• whether JSON parses
• whether citations are supported
• whether the answer remains correct at the required length

A good starting point is the broader practice described in LLM evaluations.

Structured output deserves special caution. An autoregressive decoder can emit a stable opening brace, stream fields in order, and apply grammar constraints as it goes.

A diffusion system can also be designed for constrained output, but it must maintain a partially filled structure while positions are being revised. The capability is possible; the serving and validation machinery is less forgiving.

Streaming is another dividing line. Autoregressive output can be shown as soon as each token arrives. Diffusion output may change between passes.

If the interface displays a sentence before it is stable, users see text flicker, disappear, and reappear. A product can wait for the final pass, expose only a stable prefix, or render the intermediate draft as explicitly provisional. Each choice gives back some of the apparent immediacy.

The strongest objection is fair

Autoregressive serving is mature. It has optimized kernels, quantization, continuous batching, paged KV caches, and speculative decoding.

Speculative decoding uses a smaller draft model to propose several tokens, then has the larger model verify them in fewer expensive rounds. It can reduce the practical cost of serial decoding without changing the model’s causal structure.

That matters. A new diffusion model should not be compared with a badly configured autoregressive baseline and declared the future.

If the answer is short, the existing model is higher quality, streams cleanly, and already meets the latency target, switching architectures is engineering theatre.

The objection does not erase the underlying difference. Speculative decoding can make the causal chain cheaper, but it cannot make a later token independent of the accepted earlier tokens.

Continuous batching improves aggregate throughput, but each individual sequence still advances through a dependency chain. Diffusion attacks that chain directly.

So the correct choice depends on the shape of the workload. For a 20-token customer reply, mature autoregressive serving may win.

For thousands of simultaneous 800-token drafts, a diffusion model that maintains comparable task quality may have a very different cost and latency profile. “Faster model” is not a property you can evaluate without naming the request shape.

What to do on Monday morning

Do not begin by replacing your production model. Begin by measuring whether your workload has the problem diffusion is good at solving.

1. Take 200 real prompts. Redact sensitive data, preserve the prompt lengths, and include the awkward cases. Record the output length distribution. Separate short replies from 128-token explanations, 512-token reports, and long generations. Include requests that call tools, require JSON, or must stream visibly.

2. Write down the baseline. Measure:

   • time to first token
   • time to the final token
   • p50 and p95 end-to-end latency
   • output tokens per second
   • cost per request
   • retry rate
   • task success

   The inference metrics vocabulary is useful here because “tokens per second” alone hides too much.

3. Run an apples-to-apples comparison. Keep the following fixed:

   • the prompt
   • maximum output length
   • sampling policy
   • hardware class
   • region
   • concurrency
   • timeout policy

   Test concurrency one, eight, and a higher level that resembles your real traffic. A diffusion model can look spectacular in a busy batch and ordinary for one request, or the reverse.

4. Set a quality gate before looking at speed. For the incident bot, the gate might require:

   • the correct service
   • timestamp
   • root cause
   • remediation steps

   For code, run tests. For JSON, parse it and validate its schema. A model that is 40 percent faster but causes twice as many retries is not faster at the product level.

5. Route by workload instead of ideology. Try diffusion first for long-form drafting, rewriting, infilling, and high-throughput text where a final answer matters more than a stable token stream. Keep autoregressive generation for short interactive answers, exact tool calls, strict streaming, and tasks where the model’s step-by-step reasoning is part of the design. Reasoning-heavy requests deserve their own evaluation; reasoning models are not interchangeable with fast draft generators.

6. Put a fallback behind the experiment. Set a maximum pass count and a wall-clock deadline. If the output is any of the following, send the request to the proven model:

   • incomplete
   • fails validation
   • changes too much between passes

   Track the fallback rate. Also track the number of passes actually used, because an adaptive schedule that quietly increases effort on difficult prompts can turn a good average into a bad p95.

The cost calculation should use successful work, not a vendor’s headline. Compare the full request economics with the method described in LLM cost and latency:

• model charges
• accelerator time
• queueing
• validation
• retries
• fallbacks

A half-second generation that spends three seconds waiting for a server slot is not a half-second user experience.

The failure modes show up quickly

The first symptom of a bad diffusion deployment is often not “the model is inaccurate.” It is “the dashboard says 1,000 tokens per second, but users say the app feels slow.”

Check what the dashboard starts and stops. It may exclude prompt processing and network time. It may report aggregate batch throughput. It may measure only the final pass, while difficult prompts trigger more passes. It may count generated positions even when the response is later rejected by a schema validator.

Split the trace into:

• queue time
• prompt processing
• each refinement pass
• validation
• final delivery

A second symptom is text that flashes into existence and then changes substantially. That means the interface is treating an intermediate state as a final answer.

Render only the final result, or label intermediate text as a draft. If users need trustworthy streaming, an autoregressive route is probably the simpler answer.

A third symptom is a sudden rise in truncated answers or malformed structured output. Inspect the length budget and stopping rule first.

The model may have been given too few positions, or the system may be counting mask slots differently from visible output tokens. Validate every final response and keep a fallback rather than retrying blindly.

Diffusion language models do not abolish the cost of generation. They relocate it. Autoregressive models spend their serial budget one token at a time. Diffusion models spend theirs in repeated full-sequence revisions.

For long outputs on well-utilized parallel hardware, that can be a spectacular bargain. A 1,000-plus-token-per-second vendor headline is a reason to investigate, not proof of a universal speedup.

Only a reproducible comparison that names the following can establish one:

• the model
• the hardware
• the request shape
• the pass count
• the quality target
• the throughput definition

But the right lesson is not “all LLMs should become diffusion models.” It is that left-to-right generation was a design choice, not a law of language.

When your bottleneck is a long serial chain, a model that drafts the whole shape and repairs it in parallel may be exactly the tool you needed. When your product needs a stable prefix, exact actions, or dependable short answers, the old chain may still be the better piece of engineering.