Video, 3D & audio generation

You ask a model for a ten-second clip of a red bicycle passing through a rainy alley. The first frame looks right. Three seconds later, the bicycle has acquired a second handlebar, the puddles have moved uphill, and the rider’s jacket changes colour every few frames. Add sound, and the bell may ring half a second before the rider touches it.

An image model only has to make one arrangement of pixels plausible. A video model must make many arrangements plausible together. A 3D model must make them plausible from viewpoints it has never seen. An audio model must preserve structure across 44,100 samples every second.

The useful idea is not that one magic algorithm handles everything. It does not. The useful idea is that the same broad generative families can operate on different representations:

• diffusion and flow matching transform noise into a learned representation;
• autoregression predicts the next element in a sequence;
• a decoder or renderer turns that representation into:
  • pixels;
  • geometry; or
  • sound.

The hard engineering work moves into:

• the representation;
• the conditioning;
• the constraints that connect neighbouring pieces.

The shared mental model

Suppose z is a compact representation of the thing you want to generate: a latent feature grid for an image, a space-time grid for video, codec tokens for audio, or a neural field or 3D Gaussians for a scene.

A diffusion model corrupts a training example with noise, then learns to reverse that corruption. At generation time, it starts with noise and repeatedly moves the representation toward an example matching the prompt. Latent diffusion does this in compressed space, where a decoder expands the final latent into an image or other output.

Flow matching instead trains a time-conditioned velocity field along a chosen path between a base distribution and the data distribution. Sampling follows that field with numerical ODE steps. The path and solver affect the result, but the representation remains just as important.

Autoregressive generation factorises a distribution into conditional predictions: the model predicts one token, then the next given the previous tokens. This suits text and audio, but it is serial; long sequences require many dependent predictions.

These families are not interchangeable buttons. A video diffusion transformer and an audio autoregressive transformer need different:

• tokenizers;
• attention layouts;
• data;
• decoders.

Video: consistency across time

A video is not a bag of images. Neighbouring frames constrain one another.

At 24 frames per second, a four-second clip has 96 frames. Independent sampling can satisfy the prompt 96 times while failing to preserve the same bicycle 96 times. Small texture changes become flicker; larger changes become disappearing objects, rubbery limbs, or camera jumps.

import numpy as np
rng = np.random.default_rng(0)

independent = rng.standard_normal(5).round(2).tolist()        # each frame sampled fresh -> jitter
consistent = [0.0]
for _ in range(4):
    consistent.append(round(consistent[-1] + 0.1 * rng.standard_normal(), 2))   # small step from prev

print("independent frames (flicker):", independent)
print("consistent frames (coherent):", consistent)

The model usually compresses the clip into a latent tensor and divides it into space-time patches: blocks covering several time steps and a region of the image. A transformer attends to these tokens rather than raw pixels. With 16-by-16 spatial patches, one 512-by-512 frame has 1,024 spatial tokens; across 96 frames that is 98,304 tokens before temporal compression. Full attention would consider roughly 9.66 billion token pairs. Latent compression, temporal downsampling, local attention, and chunking make generation feasible.

A video diffusion model can add noise to the whole latent clip and learn to recover clean space-time structure. Temporal attention lets a token describing a wheel use information about that wheel before and after it. The model can therefore change position and texture together instead of generating unrelated frames.

Long clips remain expensive. Systems choose among strategies such as:

• generating short shots;
• using overlapping windows;
• conditioning later windows on earlier frames;
• generating keyframes and filling in motion.

These strategies trade memory and latency against long-range consistency.

Temporal modelling helps:

• appearance;
• motion;
• geometry.

It does not guarantee physics or causality. A model can produce a smooth video of a bicycle whose wheels turn backwards. Test state changes such as braking or picking up an object; continuity is not understanding.

3D: generating a scene, not a viewpoint

A 2D image records one camera. A 3D representation must answer what the scene looks like from an unseen camera.

NeRFs: a function queried along camera rays

A NeRF, or Neural Radiance Field, represents a scene with a learned function. Given a 3D position and viewing direction, it returns a colour and density. Rendering samples many points along each camera ray and combines their contributions.

For a ray through a bicycle scene, suppose samples are 25 centimetres apart and a sample has density 4 per metre. Its probability of absorbing light over that interval is

alpha = 1 - exp(-4 × 0.25) ≈ 0.632.

That sample contributes about 63.2 percent of the light reaching it. If it is first, its visibility weight is about 0.632. The next sample receives the remaining 36.8 percent; with the same density, its weight is about 0.368 × 0.632 ≈ 0.233. The renderer adds weighted colours from many samples, with less light reaching samples farther along the ray.

This volume-rendering process makes a NeRF a continuous field, not a coloured point cloud. It is usually fitted from multiple images with known or estimated camera poses, adjusting the field until its rendered views match the observations. It produces smooth novel views and is differentiable, but repeated network queries make rendering more demanding than drawing a mesh.

Gaussian splatting: explicit primitives for fast views

3D Gaussian splatting represents a scene with soft 3D ellipsoids. Each Gaussian has:

• a position;
• shape and orientation;
• opacity;
• colour parameters.

The system projects the Gaussians into the camera, rasterizes them, and composites them by depth.

This is fast enough for interactive exploration. It is not automatically a clean mesh: floaters, holes, poor transparency, and unstable thin structures can appear, especially outside observed viewpoints. A splat is useful for a virtual tour. An editable game or CAD asset needs:

• a mesh;
• materials;
• UVs;
• collisions.

“Looks 3D” and “is an editable asset” are different deliverables.

Text-to-3D: using a 2D teacher

Paired text-and-3D data is scarce, so methods such as DreamFusion use a pretrained 2D diffusion model as a teacher:

1. Start with a parameterized 3D representation, such as a NeRF.
2. Render it from a sampled camera.
3. Ask the frozen 2D model whether the image matches the text.
4. Backpropagate a score-distillation signal into the 3D representation.
5. Repeat from many views.

The 2D model knows what “red bicycle” tends to look like, but it sees one view at a time. It does not inherently enforce that the front and back are the same object. The Janus problem is a common result: multiple sides develop front-facing interpretations. Melted geometry and missing backs are other failures.

Several techniques reduce the problem:

• multi-view supervision;
• geometric priors;
• reference images;
• better camera sampling.

They reduce the problem without turning a 2D prior into physical understanding.

Audio: a long signal with several useful representations

A ten-second clip sampled at 44.1 kilohertz contains 441,000 waveform values per channel. Direct generation must model both high-frequency detail and long-range musical or linguistic structure across that long sequence.

Representation choices

A waveform preserves everything but is lengthy.

A spectrogram exposes time-frequency structure, though a decoder must reconstruct the waveform and handle phase.

A neural audio codec compresses sound into discrete or continuous tokens that a decoder reconstructs. Coarse tokens can carry structure while fine tokens carry acoustic detail.

An autoregressive model predicts tokens one at a time:

p(z) = p(z1) × p(z2 | z1) × p(z3 | z1, z2) ...

This supports long-range structure in:

• speech;
• melody;
• lyric structure.

But generation is serial. Diffusion and flow models can refine many positions in parallel, though they typically require multiple denoising or integration steps unless distilled. The decoder still determines whether the output has clean, stable sound.

Timing and alignment

Audio timing is finer than video timing. At 24 frames per second, one frame spans about 41.7 milliseconds. A consonant, drum transient, or bell onset may be much shorter. A video can look smooth while its audio is late.

Synchronized generation needs:

• shared timing signals;
• cross-modal attention;
• explicit event alignment.

A caption alone does not put “the bell rings as the rider brakes” on a common timeline.

Choose the representation from the output contract. For example:

• a navigable capture may suit Gaussian splats or a NeRF;
• an editable game asset needs:
  • a mesh;
  • materials;
  • UVs;
  • collisions.

For sound, codec tokens or spectrograms may suit semantic generation, while precise short events demand careful timing and reconstruction.

What breaks first

• Video identity drift: logos change, clothing gains details, or faces widen across windows. Inspect consecutive frames and shot boundaries, then use reference conditioning, overlap, and shorter shots.
• Temporal wobble: static edges shimmer even when the subject moves correctly. This indicates insufficient shared temporal information or inconsistent decoder detail. Use temporal losses and inspect motion as a sequence, not a contact sheet.
• 3D floaters and Janus geometry: a scene works from one angle but fails from the side. Improve camera coverage for captured scenes; use references or multi-view supervision for text-to-3D.
• Audio clicks and smeared transients: pops at token boundaries or missing consonants often come from codec reconstruction, chunking, or weak fine-detail modelling. Inspect boundaries and use overlap-and-crossfade, hierarchical tokens, or a stronger vocoder.
• Smooth nonsense: local motion can be coherent while the requested action is impossible. Test whether causes precede effects, not merely whether frames are smooth.

As of August 2026, long-form multimodal generation is usually an assembly problem:

• generate short video shots with references and keyframes;
• regenerate failures;
• create sound with explicit timing;
• use a 3D representation suited to the final renderer.

Unified models help, but the decoder, editor, evaluator, and asset format determine whether the result can leave the demo.

Every extra axis has a cost:

• time multiplies visual tokens;
• 3D must survive viewpoint changes;
• audio demands fine timing and high-frequency fidelity.

More conditioning improves control but consumes context and can make optimisation brittle. One generative core is a useful pattern, not a claim that video, geometry, and sound are the same problem.

Quick check

Next

The machinery starts with latent diffusion and flow matching. When samples look plausible but fail in subtle ways, use the principles in generative evaluation: measure the output contract, not just whether one frame looks impressive.