Unified multimodal models
A vision-language model can see but not draw — it understands images and generates only text. Unified multimodal models do both: one model that perceives and generates images by putting everything in a single token space. Chameleon and Emu3 do pure next-token; Transfusion fuses autoregression and diffusion in one transformer. The path to any-to-any.
Written and reviewed by Shreyash Prashu Last reviewed
What you'll learn
- Why a standard VLM understands images but can't generate them
- The unified-token approach — discretize images, predict next token over everything
- The AR-vs-diffusion tension and the Transfusion hybrid
- What "any-to-any" unlocks and where it's heading
Before you start
A standard vision-language model can see but not draw — it reads images and outputs only text. Unified multimodal models close the loop: a single model that both perceives and generates images (and beyond). The trick is to stop treating images as something special and put everything in one token space.
One token space for text and images
A visual tokenizer (a VQ-VAE: a network that chops an image into patches and replaces each patch with the ID of its closest match in a fixed list of learned patterns, called a codebook) turns an image into a sequence of discrete codes. These work exactly like text tokens, except they come from an image codebook.
Now text and image are tokens in one vocabulary. A single next-token transformer (a model that repeatedly guesses the next token in a sequence) can read or write either. It switches between modalities at special marker tokens:
seq = ["<text>", "a", "red", "apple", "<image>", "img42", "img17", "img88", "</image>"]
vocab_text = {"a", "red", "apple"}
vocab_image = {"img42", "img17", "img88"} # discrete codes from a visual (VQ) tokenizer
for tok in seq:
kind = "TEXT" if tok in vocab_text else "IMAGE" if tok in vocab_image else "MARKER"
print(f"{tok:<10} -> {kind}")<text> -> MARKER
a -> TEXT
red -> TEXT
apple -> TEXT
<image> -> MARKER
img42 -> IMAGE
img17 -> IMAGE
img88 -> IMAGE
</image> -> MARKEROne autoregressive model (one that produces a sequence a token at a time, each guess conditioned on everything before it) predicts the next token across the whole sequence. So the same model that answers “what’s in this image?” can also emit image tokens to generate one. Chameleon does exactly this (early-fusion, mixed-modal tokens trained together); Emu3 pushes “just next-token prediction” over text, images, and video to its logical end.
The AR-vs-diffusion tension, and a hybrid
There’s a catch: autoregressive image tokens (from a VQ codebook) don’t match the quality of diffusion, which generates an image by starting from noise and removing it step by step. Snapping patches to a fixed codebook throws away fine detail.
So a second family combines both in one transformer. Transfusion trains with a next-token loss on text and a diffusion loss on image patches inside a single model. It gets language fluency and diffusion-grade images together. Janus takes another route — decouple the visual encoders, using one representation for understanding and a different one for generation, since the two tasks want different features. Different recipes, same goal: one model that both perceives and creates.
In one breath
- A standard VLM understands images but only generates text; unified multimodal models both perceive and generate by using one token space.
- A visual tokenizer (VQ-VAE) discretizes images into tokens, so text and image are one vocabulary and one next-token transformer reads or writes either (the demo: text + image tokens in one sequence) — Chameleon, Emu3.
- Autoregressive image tokens lose quality vs diffusion, so Transfusion fuses a next-token loss (text) and a diffusion loss (image) in one transformer; Janus decouples the understand vs generate encoders.
- This enables any-to-any (text↔image↔audio in one model) — the basis of native generation in models like GPT-4o, tighter than a bolted-on VLM + diffusion pipeline.
- The open tension is AR simplicity vs diffusion quality, with hybrids leading.
Quick check
Quick check
Next
Unified models extend the VLM architectures from understanding to generation, with diffusion often supplying the image quality. Extending the same recipe to audio and physical action is omni & embodied models.
Practice this in an interview
All questionsA multimodal vision-language model turns image regions into visual tokens, maps them into the language model's hidden representation, and combines them with text through attention. CLIP is a contrastively trained dual encoder that aligns image and text embeddings; it can provide a vision backbone or support retrieval and zero-shot classification, but it is not by itself a generative visual question-answering model.
BERT is an encoder-only transformer that reads all tokens bidirectionally and is trained with masked language modelling — ideal for tasks requiring a rich contextual representation of an entire sequence, like classification or NER. GPT is a decoder-only transformer that attends only to previous tokens via a causal mask and is trained with next-token prediction — ideal for text generation. Encoder-decoder models like T5 combine both for tasks that map one sequence to another.
Attention masks determine whether a model builds bidirectional representations, generates causally from left to right, or combines source understanding with target generation. Pretraining objectives reinforce those patterns: masked language modeling suits representations, next-token prediction suits continuation, and sequence-to-sequence reconstruction suits source-to-target tasks.
Encoder-only transformers read the whole input bidirectionally and produce representations, decoder-only transformers use causal attention to predict the next token, and encoder-decoder transformers encode a source sequence before a causal decoder generates a target sequence with cross-attention. The choice follows the task: understanding, open-ended continuation, or sequence-to-sequence transformation.