Latent diffusion & Stable Diffusion

At 3 a.m., a product team asks for 512-pixel images of “a cat in a yellow raincoat, standing in a flooded street.” You start from random noise and refine it again and again. If every refinement operates on the raw RGB image, the denoiser must process 512 × 512 × 3 = 786,432 values per example, repeatedly.

That is an expensive place to spend your compute. Most of those values describe tiny colour changes and texture. The model needs to understand a cat, a coat, and a street, but it does not need to reason over every full-resolution pixel at every denoising step.

DDPM means denoising diffusion probabilistic model: a model trained to remove controlled noise from data. Latent diffusion makes that loop cheaper by moving it into a latent space, a compact internal representation learned by another model. The architecture behind the original Stable Diffusion family combines that compressed representation with text conditioning.

The key idea is simple:

Compress once. Denoise in the compact space. Decode once.

The details explain why this works, how the prompt reaches the image, and why a guidance scale of 15 can turn a sensible cat into a glossy hallucination.

The efficiency win: denoise in a latent

A variational autoencoder, or VAE, has an encoder that compresses an image and a decoder that reconstructs it. Write the encoder as E and the decoder as D:

• z = E(x) turns an image x into a latent z.
• x̂ = D(z) turns the latent back into an image-like reconstruction x̂.

For a classic 512-pixel Stable Diffusion model, the image has shape 512 × 512 × 3. The VAE commonly maps it to a latent with shape 64 × 64 × 4:

pixel_values = 512 * 512 * 3
latent_values = 64 * 64 * 4

print(f"pixel space:  {pixel_values:,} values")
print(f"latent space: {latent_values:,} values")
print(f"-> {pixel_values // latent_values}x fewer values per denoising step")

pixel space:  786,432 values
latent space: 16,384 values
-> 48x fewer values per denoising step

The spatial width and height each shrink by 8×, while the channel count grows from 3 to 4. The total number of scalar values falls by 8 × 8 × 3 ÷ 4 = 48.

That does not mean generation is exactly 48× faster: convolutions, attention, memory movement, hardware, sampler steps, and implementation also matter. The precise advantage is that the denoiser performs repeated work on a much smaller spatial grid. The VAE decoder restores the final latent to 512-pixel output only after that iterative work.

The VAE is not a lossless zip file. Its bottleneck discards some information while preserving perceptually important structure such as edges, shapes, colours, and enough texture to look convincing. The denoiser can therefore model useful visual representations rather than every possible RGB arrangement.

What happens during training

For a training image such as our cat in a raincoat, the VAE produces a clean latent z_0. Diffusion adds noise according to a schedule:

z_t = α_t z_0 + σ_t ε

Here, t is the noise level, ε is random Gaussian noise, and α_t and σ_t determine how much clean signal and noise remain. The denoising network receives the noisy latent and timestep and learns to predict the noise, or an equivalent target such as the clean latent or a velocity.

The important boundary is that the diffusion model is trained on VAE latents, not directly on the original pixel tensor.

What happens during generation

Text-to-image generation samples random latent noise, z_T, at the desired latent resolution. A sampler uses the denoiser’s predictions to move from noisy latent to cleaner latent over a sequence of updates.

Training may define hundreds or thousands of noise levels, while inference often uses a much shorter path. A 30-step generation means 30 numerical updates selected by the sampler; it is not 30 training steps.

After the final update, the clean latent z_0 passes through the VAE decoder once, becoming a 512 × 512 × 3 image. In image-to-image generation, the input image is encoded, noise is added to its latent, and denoising starts from there. The noise amount controls how far the result can move from the source. Inpainting adds a mask so some regions remain tied to the original.

How the prompt enters the denoiser

A text prompt cannot steer the image merely by sitting beside the latent tensor. The denoiser needs a differentiable representation of the words.

A text encoder converts the prompt into a sequence of embeddings, numerical vectors whose positions capture learned relationships between tokens. Depending on the model, it may be CLIP, T5, or another language model. “A cat in a yellow raincoat” becomes several token vectors, not one vector labelled “cat”.

In the original U-Net-based Stable Diffusion architecture, those vectors enter through cross-attention. The denoiser’s image features provide queries; text embeddings provide keys and values. For one attention head:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

The dot products measure which text positions matter to each image feature. Softmax turns those scores into weights, and the weighted text values are mixed back into the image features. A latent feature near the cat’s torso might attend strongly to “yellow” and “raincoat”, while another attends to “flooded” or “street”.

This is not a one-word-per-pixel layout. Every spatial feature can consider every prompt token, repeatedly across denoising layers and timesteps. Early steps tend to establish broad composition; later steps refine edges and texture, though the boundary is not strict.

Cross-attention also does not guarantee literal instruction following. The model learned statistical associations from its training data, so “three red apples” may produce two, five, or an unclear red object.

Classifier-free guidance: turning up the prompt

Classifier-free guidance, or CFG, amplifies the difference between a prompt-conditioned prediction and an unconditioned prediction.

At one timestep, let:

• ε_uncond be the denoiser’s prediction with an empty or unconditioned prompt.
• ε_cond be its prediction with “a cat in a yellow raincoat”.
• s be the guidance scale.

The common equation is:

ε_cfg = ε_uncond + s(ε_cond − ε_uncond)

The difference ε_cond − ε_uncond is the direction contributed by the prompt. CFG extends beyond the conditional prediction rather than simply selecting it.

If a component of the unconditional prediction is 0.40 and the conditional prediction is 0.55, then at guidance scale 7:

0.40 + 7(0.55 − 0.40) = 1.45

The value 1.45 is a component of the denoising prediction, not pixel brightness. The sampler uses it to update the latent.

At s = 1, the result is the conditional prediction; at s = 0, it is the unconditional prediction. Higher values usually strengthen prompt features but can reduce variety and produce oversaturated colours, harsh edges, repeated details, or anatomical errors. More obedience is not more realism.

It is classifier-free because the model learns both conditional and unconditioned behaviour during training: some examples receive the condition and some receive none. At inference, the same network runs twice per step, once with and once without the prompt, and combines the predictions. There is no separate image classifier.

The second evaluation makes CFG’s denoiser cost close to twice that of one prediction, although implementations can concatenate both inputs into one larger batch. The VAE still decodes once.

Many interfaces call the unconditioned input a negative prompt. This is not a guaranteed list of forbidden concepts. It supplies alternative conditioning to the unconditioned branch, and its effect depends on the model, tokenisation, and guidance implementation.

The production pipeline

A practical request chooses a model, output size, seed, and sampler schedule. The seed creates the initial latent noise, so keeping it fixed makes prompt or guidance comparisons meaningful.

The text encoder creates conditioning vectors. The denoiser receives the current latent, timestep, and vectors; its prediction goes to the sampler, which calculates the next latent. After the chosen inference steps, the VAE decoder produces pixels.

Resolution changes the latent grid. With an 8× spatial reduction, a 1024-pixel image produces a 128 × 128 × 4 latent rather than 64 × 64 × 4: doubling both dimensions creates four times as many latent positions. Memory and compute rise accordingly. A model tuned for 512 pixels is not automatically reliable at arbitrary larger sizes; stretching the grid can duplicate subjects or destabilise composition.

ControlNet adds spatial conditioning such as an edge map, pose skeleton, depth map, or scribble. It processes that signal alongside diffusion features and injects the result into a locked or mostly preserved base model. The prompt can describe the cat and raincoat while a pose map specifies the cat’s head, legs, and body positions.

This solves a different problem from CFG. CFG changes how strongly the model follows semantic text; ControlNet supplies spatial evidence. It is useful for pose transfer, product layouts, and line-art conversion, at the cost of extra memory, computation, and conditioning to manage. A poor control map still gives the model poor geometry.

What fails first

Harsh halos, neon colours, or burnt shadows: lower the guidance scale and compare with the same seed. Very high CFG amplifies estimation errors as well as useful prompt direction.

Duplicated or stretched composition at larger resolution: generate at the model’s native size, then use a documented upscale or second-pass workflow. A larger canvas is not free.

Muddy colours, melted faces, or consistently wrong detail: check that the VAE matches the checkpoint and its latent scaling convention. Stable Diffusion pipelines commonly apply a model-specific scaling factor between the VAE and denoiser; use the pipeline configuration rather than copying a constant from another model.

An ignored prompt: check token truncation, the expected text encoder, and the conditioning path. Shorten the prompt to essential nouns and relations. If the problem is pose or layout, more adjectives will not create a spatial constraint; use control conditioning.

These failures reveal the trade-off. Latent diffusion is cheaper than pixel-space diffusion, but it inherits the VAE’s information bottleneck, the denoiser’s learned biases, sampler approximations, and conditioning weaknesses. It is an engineering compromise, not a free lunch hidden in a smaller tensor.

What changed after the original Stable Diffusion design

“Stable Diffusion” is a family name, not one frozen architecture. Earlier systems pair a VAE with a convolutional U-Net, text encoder, cross-attention, and CFG. Newer systems increasingly use a diffusion transformer, or DiT, which applies transformer blocks to visual latent tokens.

Some also use flow matching, an objective that learns a vector field transporting noise toward data along a chosen continuous path. It is related to diffusion sampling, but its solver, parameterisation, training target, and guidance behaviour can differ from DDPM-style systems.

The durable mental model is:

1. Represent the image in a learned space, often a latent.
2. Learn a path from noise toward realistic visual data.
3. Inject text or spatial conditions.
4. Numerically follow that path.
5. Decode the final representation into pixels.

The U-Net may become a transformer and the noise path may change, but representation, iterative prediction, conditioning, sampling, and decoding remain central.

Quick check

Next

For the underlying process, read DDPM. For the VAE bottleneck, see VAEs from scratch. The attention mechanism is developed in Attention from scratch, and the larger family is compared in Generative Models: GANs, VAEs & Diffusion.