Multimodal (vision & audio) LLMs
How multimodal models turn images, audio, and video into representations an LLM can reason over, why resolution drives token cost, and the production patterns that keep document, screen, and media systems reliable.
What you'll learn
- How a vision-language model converts image patches into inputs a language model can process
- Why resolution, tiling, and sequence length drive multimodal cost and latency
- How to build reliable pipelines for screenshots, documents, video, and audio
- When a multimodal LLM is the wrong tool for exact extraction, detection, or measurement
Before you start
At 3:07 p.m., a support engineer receives a screenshot and a sentence:
“Checkout failed. What exactly happened, and which field should I fix?”
The screenshot contains a red error banner, a tiny request ID, a browser address bar, and a payment form. A text-only LLM can answer only if another system first describes the image. That description may omit the one line that matters. OCR can read the letters, but may lose the relationship between the error and the field beneath it.
A multimodal model can inspect both the pixels and the question together. It might say that the card number is valid, the postal-code field is empty, and quote the error message. It can also confidently misread six-pixel text. The useful mental model is not “the model has eyes.” It is “a pipeline converts another kind of input into representations its language machinery can use.”
Multimodal means a model system accepts more than text, such as images, audio, or video. A vision-language model, or VLM, is the image-and-text case.
How a model “sees”
A common VLM has three conceptual stages:
- A vision encoder converts pixels into vectors.
- A connector or projection layer adapts those vectors to the language model.
- The language model processes visual and text representations together.
The vision encoder usually starts with patches. A 224 by 224 image with 14 by 14 patches has:
224 / 14 = 16 patches across, and 16 × 16 = 256 patches total.
Each patch contains 14 × 14 × 3 = 588 colour values. A learned layer turns those values into a continuous vector. The encoder lets patches exchange information, so a wheel can be interpreted differently when surrounding patches suggest a car rather than a bicycle.
This resembles text tokenization only in broad outline. A text tokenizer maps words to discrete vocabulary IDs. An image patch usually becomes a floating-point vector such as [-0.14, 0.82, ...]. “Image token” is useful shorthand, not a claim that every possible patch has a vocabulary entry.
The connector maps the encoder’s output to the LLM’s expected width. If the vision encoder produces vectors of width 1,024 and the LLM expects 4,096, the connector changes that width while training visual patterns to align with concepts the language model can use.
Conceptually, the LLM receives one sequence:
[text: "What happened?"] [image vectors] [text: "Quote the error exactly."]Attention can connect “error” to text-like visual regions and “which field” to the banner and form controls below it. The model still generates one text token at a time; the input simply contains another kind of representation. See the autoregressive loop.
Other architectures use cross-attention instead of placing every visual vector in the text sequence, or use a resampler to compress patch features. The practical question is how many representations the input becomes and whether they preserve the detail the task needs.
Why images consume tokens
A 512 by 512 image with 14-pixel patches would produce roughly:
ceil(512 / 14) × ceil(512 / 14) = 37 × 37 = 1,369
patch positions if processed without resizing. An API may produce a different number because it resizes, tiles, merges patches, or adds high-resolution crops. “One image equals one token” is a dangerous simplification.
Many production encoders use square tiles. For a 1,600 by 900 screenshot with 512-pixel tiles:
ceil(1600 / 512) × ceil(900 / 512) = 4 × 2 = 8 tiles
JPEG compression changes the bytes sent over the network, not necessarily the number of visual tokens processed.
There are two costs:
- Input token cost for visual and text representations.
- Output token cost for generating the answer.
More visual tokens also increase prefill work: the model must read the complete input before producing its first output. Transformer attention compares tokens with one another, so a longer visual sequence can increase computation faster than a one-for-one count suggests. The context window is another constraint; a large window does not make thousands of visual tokens cheap or useful.
The support screenshot illustrates a useful trade-off. Send a 1,024 by 576 overview first: 4 tiles in the representative tiled calculation. If the decisive text is in a 512 by 256 error-banner crop, send that crop second: 1 more tile. The combined estimate is:
2 × 85 + 5 × 256 = 1,450 tokens
Sending the full 1,600 by 900 screenshot would be about:
85 + 8 × 256 = 2,133 tokens
The overview-plus-crop approach uses fewer illustrative tokens and gives the error text more pixels per token. It is not always better: separate images add another image marker and can confuse the model unless the crop is labelled. Spend resolution where the answer lives.
A production pipeline
1. Define the visual task
“Understand this image” is not a requirement. Decide whether the system needs a description, exact text, a classification, a comparison, a location, a measurement, or an action. The task determines preprocessing and validation.
For the checkout screenshot, request structured output containing the quoted error, suspected field, and uncertainty flag. Requiring a quote exposes whether the model read the pixels or invented a plausible explanation.
2. Make a cheap overview
Start at a reasonable size and preserve the aspect ratio. Remove irrelevant browser chrome from screenshots, but retain enough resolution to distinguish document columns or table rows. A thumbnail may identify a bicycle; it will not reliably read a serial number.
Let the overview locate the relevant region, then inspect that region at higher resolution. This coarse-to-fine strategy saves cost, but an overview can miss tiny, low-contrast, or unusual targets. For absence claims or high-stakes extraction, use full-resolution OCR or tiling, multiple candidate crops, a detector, or human review. Treat an overview-only “not found” as inconclusive.
3. Crop and label detail
Crop a relevant region with enough surrounding context to preserve relationships. For exact extraction, use a narrow second-pass instruction:
Read the error message exactly as displayed.
Return the text in quotation marks.
If any character is unclear, use [unclear] rather than guessing.When sending several images, label them: “Image 1 is the original screen. Image 2 is a crop of the error banner.” For comparisons, state which is before and after.
4. Validate consequential fields
A fluent answer is not evidence. Use schemas and constrained decoding to enforce output shape, but remember that neither proves digits are correct. Check arithmetic on extracted totals, image bounds on claimed coordinates, and high-value fields with a second pass, deterministic OCR, or human review.
Treat coordinates as suggestions until a separate check confirms that the target exists and the action is safe.
Documents, video, and audio
Documents combine text with layout. A VLM can reason over tables, handwriting, visual marks, and relationships between regions. It is not a universal OCR replacement: for millions of clean invoices, conventional extraction is cheaper, faster, deterministic, and easier to audit. Use a VLM for messy scans, unusual layouts, or questions that require interpretation.
Video is a sequence of images plus time and often speech. A 60-second clip at 30 frames per second contains 1,800 frames. A practical pipeline combines a transcript, sparse frames, extra frames around cuts or motion, and timestamps. One frame per second may capture scene context but miss a sub-second event, so sampling is a cost control, not temporal coverage.
Audio can use speech-to-text or a native audio-capable model. Transcription is usually best for meeting search. Native audio matters when tone, pauses, overlapping speakers, music, or non-speech sounds affect the answer. A 10-second recording divided into 20-millisecond windows starts with:
10 / 0.020 = 500
analysis frames. These are usually compressed before reaching the LLM and are not necessarily 500 billable tokens.
Choosing the right tool
| Situation | Start with | Escalate when |
|---|---|---|
| Explain a screenshot | Multimodal LLM | Exact coordinates or repeated actions need a detector and validation |
| Extract a clean invoice | OCR plus layout parsing | Scans, handwriting, or unusual table relationships defeat it |
| Read tiny serial numbers | High-resolution crop plus OCR | The crop remains ambiguous; require human review |
| Summarise a meeting | Speech-to-text plus text LLM | Tone, overlap, or non-speech events matter |
| Find defects repeatedly | Vision classifier or detector | Novel defects or explanations require multimodal reasoning |
| Understand short video | Transcript plus sampled frames | Fast events require motion-triggered or denser sampling |
Use the LLM where ambiguity and explanation are valuable. Use a specialised system where repeatability, exactness, throughput, or measurement is the product.
Multimodal inputs also create privacy risks. Screenshots may contain credentials, audio may include bystanders, and video captures people outside the task. Redact sensitive regions, strip metadata when appropriate, define retention, and send only the necessary scene.
What fails first
Tiny text is misread confidently. Downscale less, crop the region, request verbatim transcription with uncertainty markers, and verify important values with OCR or a human.
The model answers the wrong region or mixes images. Label images, crop aggressively, and request a short evidence quote or per-image observation before the conclusion.
Document layout is flattened. OCR may extract every word while losing row and column relationships. Preserve coordinates and page boundaries, use layout-aware extraction, and validate totals.
Instructions inside an image are treated as commands. Text in screenshots, PDFs, web pages, and transcripts is untrusted data, not authority. Keep tool permissions outside the model’s interpretation and apply prompt-injection defences.
Video sampling misses events. Add transcripts, scene-change detection, motion-triggered frames, or a second pass around suspicious timestamps. A sparse summary is not proof that an event did not occur.
In one breath
Encoders turn pixels, audio, or video into vectors; connectors or cross-attention make them usable by the language model. More visual detail means more representations, prefill work, latency, and often cost. Define the task, use a modest overview, crop selectively, request evidence, and validate consequential fields. Choose OCR, detectors, transcripts, or human review when exactness matters more than flexible explanation.
LLM cost and latency and inference metrics apply to visual tokens too. For machine-checked outputs, see structured outputs.
Quick check
Quick check
Next
Images are tokens, not text. They have their own preprocessing, resolution trade-offs, and failure modes. Start with RAG basics when the missing information is in documents, and use guardrails when a multimodal answer can trigger an action.
Practice this in an interview
All questionsA multimodal vision-language model turns image regions into visual tokens, maps them into the language model's hidden representation, and combines them with text through attention. CLIP is a contrastively trained dual encoder that aligns image and text embeddings; it can provide a vision backbone or support retrieval and zero-shot classification, but it is not by itself a generative visual question-answering model.
Hallucinations occur because an LLM is trained to produce plausible next tokens, not verified facts — it has no internal truth-checking mechanism, only statistical patterns. Common causes include rare or conflicting training data, overconfident decoding, and prompts that lead the model to extrapolate beyond what it learned. Mitigation strategies include retrieval-augmented generation, grounding responses to retrieved sources, lowering temperature, and calibrated refusal training.
A token is the smallest unit a language model processes — typically a word, sub-word fragment, or punctuation mark produced by a byte-pair encoding (BPE) or similar algorithm. Pricing is per token because each token requires one forward-pass position in the attention matrix, directly driving compute and memory cost regardless of whether it maps to a full word or a single letter.
An LLM generates text one token at a time by computing a probability distribution over its entire vocabulary for the next token, sampling from that distribution, appending the result, and repeating — a process called autoregression. Each new token is conditioned on all previously generated tokens, so the output at step N is only as good as the choices made at steps 1 through N-1.