VLM model families

New vision-language models ship constantly — Qwen-VL, InternVL, Pixtral, Llama-Vision, and on. Trying to memorize them is hopeless and pointless. The useful move is to learn the handful of axes that distinguish VLM architectures in practice. Get the axes, and any new model drops into place.

Resolution: the axis that mattered most

Early VLMs forced every image to a fixed, low resolution (often 224×224). For a photo of a cat, fine. For a dense document, a chart, or a screenshot full of small text, catastrophic — the detail is simply gone before the model sees it. The biggest jump in VLM capability came from any-resolution handling: tile a high-resolution image into pieces, encode each, and produce more visual tokens for more detail. Token count scales with resolution:

patch = 14                                   # ViT patch size

for (w, h) in [(224, 224), (448, 448), (896, 448)]:
    n_tokens = (w // patch) * (h // patch)
    print(f"{w}x{h} image -> {n_tokens} visual tokens")

224x224 image -> 256 visual tokens
448x448 image -> 1024 visual tokens
896x448 image -> 2048 visual tokens

More resolution means more visual tokens means the model can read fine detail — but also more compute and context per image. That’s the central trade-off of modern VLMs: enough tokens to read the chart, not so many that the LLM drowns. Dynamic tiling (and native arbitrary-resolution encoders) let a model spend tokens where the image needs them.

The other axes

• Native-multimodal vs bolted-on. Bolted-on VLMs (LLaVA-style) connect a pretrained vision encoder to a pretrained LLM — cheap and effective, but the two were trained apart. Native-multimodal models are trained on text and images together from the start (often a unified token space), giving tighter integration and stronger reasoning at much higher training cost. The field is trending native.
• The vision encoder. Which one — CLIP ViT, SigLIP, or a stronger/custom encoder — and how many visual tokens it emits, sets the perception ceiling.
• Scale and data. A bigger LLM backbone and more (and cleaner) interleaved image-text data, including OCR, charts, and grounding data, lift everything.

In one breath

• Dozens of VLMs reduce to a few axes; learn the axes, not the brands.
• Resolution mattered most: early VLMs forced fixed low res (224×224), losing detail in documents/charts; any-resolution tiling emits more visual tokens for more detail (the demo: 256 → 2048) — at more compute and context.
• Native-multimodal (trained on text+images together) integrates tighter than bolted-on (connect a pretrained encoder to a pretrained LLM), at higher cost — the trend is native.
• Other axes: the vision encoder (CLIP/SigLIP/custom, token count) sets the perception ceiling, and scale + data (OCR, charts, grounding) lift everything.
• The axes map to capabilities — OCR/documents, region grounding, video — so choose by capability, and read any new VLM as a point in this small space.

Quick check

Next

These families all build on the VLM architectures (how vision connects to the LLM) and a ViT/CLIP encoder. The capability frontier extends to unified any-to-any models, video, and document understanding.