Vision-language model architectures

CLIP lets you compare images and text, but a vision-language model (VLM) does more: it lets an LLM see — describe an image, answer questions about it, read a chart, reason over a screenshot. The whole design problem is one question: a ViT outputs image features, an LLM consumes tokens — how do you connect them? There are three classic answers.

LLaVA: project image features into visual tokens

The simplest and now most common approach (LLaVA): take a frozen vision encoder’s patch features, pass them through a small projection (a linear layer or MLP) that maps them into the LLM’s embedding dimension, and prepend the results to the text tokens as “visual tokens.” The LLM then attends to them exactly like words:

import numpy as np
rng = np.random.default_rng(0)
llm_dim = 8

image_features = rng.standard_normal((4, 16))      # 4 patches, vision-encoder dim 16
projection     = rng.standard_normal((16, llm_dim))# learned linear: vision-dim -> LLM-dim
visual_tokens  = image_features @ projection        # 4 visual tokens, now in LLM space
text_tokens    = rng.standard_normal((3, llm_dim))  # 3 word tokens (already in LLM space)
sequence       = np.vstack([visual_tokens, text_tokens])

print(f"image -> {visual_tokens.shape[0]} visual tokens of dim {llm_dim}")
print(f"text  -> {text_tokens.shape[0]} text tokens of dim {llm_dim}")
print(f"LLM input: {sequence.shape[0]} tokens (visual + text) -> the LLM attends to both alike")

image -> 4 visual tokens of dim 8
text  -> 3 text tokens of dim 8
LLM input: 7 tokens (visual + text) -> the LLM attends to both alike

That’s the entire trick: a projection turns the image into tokens, they sit in the same sequence as the text, and the LLM’s self-attention does the cross-modal reasoning. LLaVA trains this with visual instruction tuning — image question-answer data (much of it generated by a stronger model) — and mostly just trains the small projector (and optionally the LLM), keeping the vision encoder frozen.

BLIP-2 and Flamingo: two other bridges

• BLIP-2 (Q-Former). A lightweight Querying Transformer sits between a frozen image encoder and a frozen LLM. A small set of learned query tokens attend to the image features and extract the most relevant information into a fixed, small number of tokens. Keeping both big models frozen makes training cheap.
• Flamingo (gated cross-attention). Instead of prepending tokens, insert new cross-attention layers into the (frozen) LLM that attend to the image features. They’re gated — initialized as near no-ops so they don’t break the pretrained LLM — and it naturally handles interleaved image-and-text sequences.

In one breath

• A VLM lets an LLM see; the design question is how to feed image features (from a ViT) into a token-consuming LLM — three classic answers.
• LLaVA (projection): an MLP maps patch features into the LLM’s embedding space as visual tokens, prepended to the text; the LLM’s self-attention does the cross-modal reasoning (the demo: 4 visual + 3 text → one 7-token sequence). Trained via visual instruction tuning, mostly the small projector.
• BLIP-2 (Q-Former): learned query tokens distill the image into a few tokens between a frozen encoder and frozen LLM — cheap to train.
• Flamingo (gated cross-attention): insert gated cross-attention layers into the LLM that attend to image features, handling interleaved image-text.
• The projection approach won for simplicity and scaling — turn the image into tokens, drop them in the sequence, let the LLM reason.

Quick check

Next

VLMs build on the CLIP image encoder and ViT, reusing the LLM’s self-attention for fusion. How specific model families differ — resolution, native-multimodal — is VLM model families.