Vision-language model architectures

CLIP lets you compare images and text, but a vision-language model (VLM) does more: it lets an LLM see — describe an image, answer questions about it, read a chart, reason over a screenshot. The whole design problem is one question: a ViT (a vision transformer, which splits an image into patches and turns each into a list of numbers called a feature vector) outputs image features, while an LLM consumes tokens. How do you connect them? There are three classic answers.

LLaVA: project image features into visual tokens

The simplest and now most common approach is LLaVA. Take the patch features from a frozen vision encoder (one whose weights are not updated during training). Pass them through a small projection, meaning a linear layer or a small MLP (a few stacked linear layers). The projection maps them into the LLM’s embedding dimension, the size of the number-list the LLM uses for each token. Then prepend (place at the start) the results to the text tokens as “visual tokens.” The LLM attends to them exactly like words:

import numpy as np

rng = np.random.default_rng(0)
llm_dim = 8

image_features = rng.standard_normal((4, 16))      # 4 patches, vision-encoder dim 16
projection     = rng.standard_normal((16, llm_dim))# learned linear: vision-dim -> LLM-dim
visual_tokens  = image_features @ projection        # 4 visual tokens, now in LLM space
text_tokens    = rng.standard_normal((3, llm_dim))  # 3 word tokens (already in LLM space)
sequence       = np.vstack([visual_tokens, text_tokens])

print(f"image -> {visual_tokens.shape[0]} visual tokens of dim {llm_dim}")
print(f"text  -> {text_tokens.shape[0]} text tokens of dim {llm_dim}")
print(f"LLM input: {sequence.shape[0]} tokens (visual + text) -> the LLM attends to both alike")

image -> 4 visual tokens of dim 8
text  -> 3 text tokens of dim 8
LLM input: 7 tokens (visual + text) -> the LLM attends to both alike

That’s the entire trick: a projection turns the image into tokens, they sit in the same sequence as the text, and the LLM’s self-attention does the cross-modal reasoning. LLaVA trains this with visual instruction tuning — image question-answer data (much of it generated by a stronger model) — and mostly just trains the small projector (and optionally the LLM), keeping the vision encoder frozen.

BLIP-2 and Flamingo: two other bridges

• BLIP-2 (Q-Former). A lightweight Querying Transformer sits between a frozen image encoder and a frozen LLM. A small set of learned query tokens (placeholder vectors the model learns during training, which act like questions asked of the image) attend to the image features. They pull the most relevant information into a fixed, small number of tokens. Keeping both big models frozen makes training cheap.
• Flamingo (gated cross-attention). Instead of prepending tokens, insert new cross-attention layers into the (frozen) LLM. Cross-attention is attention where the text positions look at a different source, here the image features, instead of at each other. The layers are gated: a learned dial starts at zero, so they begin as near no-ops and don’t break the pretrained LLM. This design naturally handles interleaved image-and-text sequences.

In one breath

• A VLM lets an LLM see; the design question is how to feed image features (from a ViT) into a token-consuming LLM — three classic answers.
• LLaVA (projection): an MLP maps patch features into the LLM’s embedding space as visual tokens, prepended to the text; the LLM’s self-attention does the cross-modal reasoning (the demo: 4 visual + 3 text → one 7-token sequence). Trained via visual instruction tuning, mostly the small projector.
• BLIP-2 (Q-Former): learned query tokens distill the image into a few tokens between a frozen encoder and frozen LLM — cheap to train.
• Flamingo (gated cross-attention): insert gated cross-attention layers into the LLM that attend to image features, handling interleaved image-text.
• The projection approach won for simplicity and scaling — turn the image into tokens, drop them in the sequence, let the LLM reason.

Quick check

Next

VLMs build on the CLIP image encoder and ViT, reusing the LLM’s self-attention for fusion. How specific model families differ — resolution, native-multimodal — is VLM model families.