Video, 3D & audio generation
Image generation is largely solved; the frontier extends the same diffusion and flow machinery along three new axes. Video adds time and needs temporal consistency. 3D adds space via neural fields and Gaussian splatting. Audio adds sound. Same generative core, a new dimension each — the path toward world models.
What you'll learn
- Text-to-video and the temporal-consistency challenge
- 3D generation — NeRF, Gaussian splatting, and text-to-3D
- Audio generation with the same diffusion/AR families
- The common thread — one generative core, a new axis per modality
Before you start
Diffusion and flow matching cracked image generation. The frontier is extending that same machinery along three new axes: time (video), space (3D), and sound (audio). Each adds a dimension — and a new challenge — to a generative core you already understand.
Video: the temporal-consistency problem
A video is frames over time, so the naive approach — generate each frame independently — produces flicker: an object’s appearance and position jump around randomly. The whole game is temporal consistency: frames must cohere so motion is smooth and objects persist.
import numpy as np
rng = np.random.default_rng(0)
independent = rng.standard_normal(5).round(2).tolist() # each frame sampled fresh -> jitter
consistent = [0.0]
for _ in range(4):
consistent.append(round(consistent[-1] + 0.1 * rng.standard_normal(), 2)) # small step from prev
print("independent frames (flicker):", independent)
print("consistent frames (coherent):", consistent)independent frames (flicker): [0.13, -0.13, 0.64, 0.1, -0.54]
consistent frames (coherent): [0.0, 0.04, 0.17, 0.26, 0.19]The independent “positions” jump all over; the consistent ones move in smooth small steps. Real text-to-video models enforce this with temporal attention (attending across frames) and by treating the video as space-time patches — Sora-style diffusion transformers diffuse over the whole spatio-temporal volume at once, so motion is generated coherently rather than frame-by-frame.
3D and audio: same core, new representation
- 3D. A NeRF (Neural Radiance Field) represents a scene as a function
(position, viewing direction) -> (color, density), rendered by integrating along camera rays — a learned 3D scene. Gaussian splatting is the faster, explicit alternative (a cloud of 3D Gaussians, real-time to render). Text-to-3D (DreamFusion) cleverly uses a frozen 2D diffusion model to supervise a 3D representation, distilling 2D priors into 3D. - Audio. Music, speech, and sound effects use the same families — autoregressive over discrete audio tokens (AudioLM, Suno-style music) or diffusion over spectrograms/waveforms — just a different representation of the signal.
In one breath
- Image generation’s diffusion/flow machinery extends along three axes: time (video), space (3D), sound (audio).
- Video needs temporal consistency — independent frames flicker (the demo), so models use temporal attention and space-time patches (Sora-style diffusion transformers) to generate coherent motion.
- 3D: NeRF learns a scene as
(position, direction) -> (color, density); Gaussian splatting is the fast explicit version; text-to-3D (DreamFusion) distills a 2D diffusion prior into 3D. - Audio: same autoregressive (audio tokens) or diffusion (spectrograms) families, new representation.
- The core is modality-agnostic — change the representation, keep diffusion/flow/AR — pointing toward video world models and unified audio-visual-3D generation.
Quick check
Quick check
Next
These extend latent diffusion and flow matching to new modalities, evaluated with generative metrics. Video as a world model connects to the broader frontier of simulators and embodied AI.