Latent diffusion & Stable Diffusion

DDPM denoises in pixel space, and a 512×512 image is a huge tensor to run a U-Net over a thousand times. Latent diffusion — the architecture behind Stable Diffusion — made text-to-image practical with one key move: run the entire noising/denoising process inside a compressed latent space instead of on raw pixels, and inject the text prompt through cross-attention.

The efficiency win: denoise in a latent

A pretrained VAE encoder compresses the image into a small latent (a downsampled, channel-rich grid); diffusion runs there; then the VAE decoder turns the final latent back into a full-resolution image — once, at the very end. The U-Net never touches full-res pixels during the expensive iterative loop:

pixel  = 512 * 512 * 3        # raw image
latent = 64 * 64 * 4          # VAE-compressed latent (8x spatial downsample)

print(f"pixel space:  {pixel:,} dims")
print(f"latent space: {latent:,} dims")
print(f"-> {pixel // latent}x fewer dims: the U-Net denoises a {pixel // latent}x smaller tensor each step")

pixel space:  786,432 dims
latent space: 16,384 dims
-> 48x fewer dims: the U-Net denoises a 48x smaller tensor each step

A 48× smaller tensor, a thousand times — that’s the difference between needing a datacenter and running on a consumer GPU. The VAE handles the perceptual detail; the diffusion model only has to learn the semantic structure in the compact latent.

The pipeline, and text conditioning

Stable Diffusion wires together three networks, with the prompt steering the denoiser:

The prompt is turned into embeddings by a text encoder (CLIP or T5), then injected into the U-Net through cross-attention at every layer: the denoiser’s image features form the queries, the text embeddings the keys/values, so each step the model attends to the words and steers the latent toward them. That’s how “a cat in a hat” pulls the noise toward that picture.

Classifier-free guidance (CFG) sharpens the effect: run the denoiser with and without the prompt, then extrapolate away from the unconditional prediction. A guidance scale dials how hard the image is pushed toward the prompt — higher means stronger adherence (at some cost to diversity and realism).

In one breath

• Latent diffusion runs the DDPM process inside a pretrained VAE latent instead of pixels — the demo’s 48× fewer dims is the difference between a datacenter and a consumer GPU.
• The VAE encodes to a latent, the U-Net denoises there for T steps, and the VAE decodes once at the end — the expensive loop never touches full-res pixels.
• The text prompt is embedded (CLIP/T5) and injected via cross-attention (image features = queries, text = keys/values), so the denoiser attends to the words and steers toward them.
• Classifier-free guidance runs with/without the prompt and extrapolates away from unconditional; a guidance scale trades prompt adherence against diversity.
• Stable Diffusion = VAE + U-Net + text cross-attention + CFG; ControlNet adds spatial control; the frontier shifts to diffusion transformers (DiT) and flow matching with the same recipe.

Quick check

Next

Latent diffusion stacks a VAE under DDPM and conditions via cross-attention; the family is framed in the generative-models overview. The frontier moves toward flow matching and diffusion transformers.