Multimodal RAG

RAG (retrieval-augmented generation, where an LLM answers using documents fetched for the question) retrieves text chunks. But a huge share of real knowledge lives in charts, tables, scanned PDFs, slides, and diagrams. The classic pipeline runs OCR (software that turns a picture of text into text), then parses the result, then chunks the text. That pipeline flattens everything, dropping the layout and figures that carry the meaning. Multimodal RAG retrieves visual content directly.

Vision-native retrieval: skip the OCR

The key technique is vision-native retrieval (ColPali-style): embed the document page image itself into a space shared with the text query, with no OCR and no parsing. An embedding is a list of numbers that captures meaning, and a shared space means text and images can be compared directly. A CLIP-like or VLM encoder (a vision-language model, one that reads images and text together) reads the page as an image. A chart’s shape and a table’s structure then survive into the embedding. Retrieval is ordinary similarity search, but over page images:

import numpy as np

query = np.array([0.9, 0.1])                       # "revenue chart" query embedding
pages = {"p1 cover": [0.1, 0.9], "p2 revenue chart": [0.85, 0.15], "p3 appendix": [0.4, 0.6]}

def cos(a, b):
    a, b = np.array(a), np.array(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

for name in sorted(pages, key=lambda n: cos(query, pages[n]), reverse=True):
    print(f"{cos(query, pages[name]):.2f}  {name}")

1.00  p2 revenue chart
0.64  p3 appendix
0.22  p1 cover

The query retrieves the revenue-chart page straight from its image — no OCR ever ran, and the chart (which OCR would have mangled into garbled numbers) is matched as a visual object. This both simplifies the pipeline (delete the brittle OCR/layout-parsing stage) and preserves information text extraction loses.

Cross-modal retrieval and multimodal agents

Because images and text share a space, retrieval is cross-modal: a text query can fetch images (and vice versa), so “find the slide with the Q3 funnel diagram” just works. Then the retrieved visual content is handed to a VLM, which reads the chart or screenshot to ground its answer — a multimodal agent that reasons over figures, not just extracted text. The full loop: embed pages as images → retrieve cross-modally → a VLM reads the retrieved pages → grounded answer.

In one breath

• Standard RAG retrieves text, but its OCR → parse pipeline flattens charts, tables, and layout — losing the meaning in visual documents.
• Vision-native retrieval (ColPali) embeds the page image itself into a space shared with the query (no OCR), so retrieval is similarity search over images (the demo retrieves the revenue-chart page from its picture).
• This simplifies the pipeline (delete OCR/parsing) and preserves information text extraction drops.
• Because image and text share a space, retrieval is cross-modal (text query → image), and a VLM reads the retrieved page to ground its answer — a multimodal agent over figures.
• Use it for chart/scan/slide/diagram-heavy content; for plain prose, cheaper text chunking still suffices.

Quick check

Next

Multimodal RAG combines RAG with the shared CLIP space and a VLM reader; it’s the visual counterpart to text chunking and advanced RAG. Reading the retrieved pages connects to document understanding.