Video & document understanding

A vision-language model handles a single image comfortably. Two high-value domains break that comfort: video (a sequence of many frames) and dense documents (forms, tables, charts packed with small text). Both come down to the same constraint — a finite visual-token budget against a great many pixels — but they spend it differently.

Video: too many frames, so sample

A video is just frames over time, and feeding all of them is hopeless. At ~256 tokens per frame, even a short clip overruns any context:

fps, seconds, tpf = 30, 60, 256
total = fps * seconds
print(f"60s @ {fps}fps = {total} frames -> {total*tpf:,} visual tokens (far over budget)")

budget = 4096
keep = budget // tpf
print(f"token budget {budget} -> sample {keep} frames (1 every {total//keep}) = {keep*tpf} tokens")

60s @ 30fps = 1800 frames -> 460,800 visual tokens (far over budget)
token budget 4096 -> sample 16 frames (1 every 112) = 4096 tokens

A one-minute clip wants 460,800 visual tokens — wildly over budget — so you sample frames down to what fits (here, 16). Beyond uniform sampling, systems use keyframe selection, token compression/pooling per frame, and memory for long video. And video needs temporal modeling — understanding motion and order across frames (“the person picks up then drops the cup”), not just what’s in each frame independently. The frontier is hour-long video, where smart frame selection and compression matter most.

Documents: too much detail, so go high-resolution and OCR-free

Documents fail the other way: not too many images, but too much fine detail in one. The fix is the any-resolution capability — tile the page at high resolution so the model can read small text, tables, and chart labels. The bigger shift is OCR-free understanding: modern VLMs read a document end-to-end from the image — layout, tables, charts, handwriting, and all — rather than running a separate OCR→parse pipeline that flattens structure and drops figures. One model sees the page the way a person does.

In one breath

• A VLM handles one image; video (many frames) and dense documents (much detail) stress the same limit — a finite visual-token budget.
• Video overruns instantly (the demo: 60s → 460,800 tokens), so you sample frames to fit (16 here), plus keyframe selection, token compression, memory; and you need temporal modeling (motion/order, not just per-frame content).
• Documents fail on detail, fixed by high-resolution any-resolution tiling so small text is legible.
• The shift is OCR-free: modern VLMs read documents end-to-end from the image (layout, tables, charts) instead of an OCR→parse pipeline that drops structure.
• Both are one discipline: spend the token budget wisely — over time (video) or space (documents).

Quick check

Next

These push the VLM model families’ resolution and context to their limits, on the ViT token budget. Retrieving visual content to feed a VLM is multimodal RAG; video also connects to omni models.