Unified multimodal models

A standard vision-language model can see but not draw — it reads images and outputs only text. Unified multimodal models close the loop: a single model that both perceives and generates images (and beyond). The trick is to stop treating images as something special and put everything in one token space.

One token space for text and images

A visual tokenizer (a VQ-VAE) discretizes an image into a sequence of discrete codes — exactly like text tokens, just from an image codebook. Now text and image are tokens in one vocabulary, and a single next-token transformer can read or write either, switching modality at special markers:

seq = ["<text>", "a", "red", "apple", "<image>", "img42", "img17", "img88", "</image>"]
vocab_text  = {"a", "red", "apple"}
vocab_image = {"img42", "img17", "img88"}      # discrete codes from a visual (VQ) tokenizer

for tok in seq:
    kind = "TEXT" if tok in vocab_text else "IMAGE" if tok in vocab_image else "MARKER"
    print(f"{tok:<10} -> {kind}")

<text>     -> MARKER
a          -> TEXT
red        -> TEXT
apple      -> TEXT
<image>    -> MARKER
img42      -> IMAGE
img17      -> IMAGE
img88      -> IMAGE
</image>   -> MARKER

One autoregressive model predicts the next token across the whole sequence — so the same model that answers “what’s in this image?” can also emit image tokens to generate one. Chameleon does exactly this (early-fusion, mixed-modal tokens trained together); Emu3 pushes “just next-token prediction” over text, images, and video to its logical end.

The AR-vs-diffusion tension, and a hybrid

There’s a catch: autoregressive image tokens (from a VQ codebook) don’t match the quality of diffusion — discretizing pixels loses fidelity. So a second family combines both in one transformer: Transfusion runs a next-token loss on text and a diffusion loss on image patches inside a single model, getting language fluency and diffusion-grade images together. Janus takes another route — decouple the visual encoders, using one representation for understanding and a different one for generation, since the two tasks want different features. Different recipes, same goal: one model that both perceives and creates.

In one breath

• A standard VLM understands images but only generates text; unified multimodal models both perceive and generate by using one token space.
• A visual tokenizer (VQ-VAE) discretizes images into tokens, so text and image are one vocabulary and one next-token transformer reads or writes either (the demo: text + image tokens in one sequence) — Chameleon, Emu3.
• Autoregressive image tokens lose quality vs diffusion, so Transfusion fuses a next-token loss (text) and a diffusion loss (image) in one transformer; Janus decouples the understand vs generate encoders.
• This enables any-to-any (text↔image↔audio in one model) — the basis of native generation in models like GPT-4o, tighter than a bolted-on VLM + diffusion pipeline.
• The open tension is AR simplicity vs diffusion quality, with hybrids leading.

Quick check

Next

Unified models extend the VLM architectures from understanding to generation, with diffusion often supplying the image quality. Extending the same recipe to audio and physical action is omni & embodied models.