Multimodal RAG

RAG retrieves text chunks. But a huge share of real knowledge lives in charts, tables, scanned PDFs, slides, and diagrams — and the classic pipeline (OCR → parse → chunk the text) flattens all of it, dropping the very layout and figures that carry the meaning. Multimodal RAG retrieves visual content directly.

Vision-native retrieval: skip the OCR

The key technique is vision-native retrieval (ColPali-style): embed the document page image itself into a space shared with the text query — no OCR, no parsing. A CLIP-like or VLM encoder reads the page as an image, so a chart’s shape and a table’s structure survive into the embedding. Retrieval is then ordinary similarity search, but over page images:

import numpy as np

query = np.array([0.9, 0.1])                       # "revenue chart" query embedding
pages = {"p1 cover": [0.1, 0.9], "p2 revenue chart": [0.85, 0.15], "p3 appendix": [0.4, 0.6]}

def cos(a, b):
    a, b = np.array(a), np.array(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

for name in sorted(pages, key=lambda n: cos(query, pages[n]), reverse=True):
    print(f"{cos(query, pages[name]):.2f}  {name}")

1.00  p2 revenue chart
0.64  p3 appendix
0.22  p1 cover

The query retrieves the revenue-chart page straight from its image — no OCR ever ran, and the chart (which OCR would have mangled into garbled numbers) is matched as a visual object. This both simplifies the pipeline (delete the brittle OCR/layout-parsing stage) and preserves information text extraction loses.

Cross-modal retrieval and multimodal agents

Because images and text share a space, retrieval is cross-modal: a text query can fetch images (and vice versa), so “find the slide with the Q3 funnel diagram” just works. Then the retrieved visual content is handed to a VLM, which reads the chart or screenshot to ground its answer — a multimodal agent that reasons over figures, not just extracted text. The full loop: embed pages as images → retrieve cross-modally → a VLM reads the retrieved pages → grounded answer.

In one breath

• Standard RAG retrieves text, but its OCR → parse pipeline flattens charts, tables, and layout — losing the meaning in visual documents.
• Vision-native retrieval (ColPali) embeds the page image itself into a space shared with the query (no OCR), so retrieval is similarity search over images (the demo retrieves the revenue-chart page from its picture).
• This simplifies the pipeline (delete OCR/parsing) and preserves information text extraction drops.
• Because image and text share a space, retrieval is cross-modal (text query → image), and a VLM reads the retrieved page to ground its answer — a multimodal agent over figures.
• Use it for chart/scan/slide/diagram-heavy content; for plain prose, cheaper text chunking still suffices.

Quick check

Next

Multimodal RAG combines RAG with the shared CLIP space and a VLM reader; it’s the visual counterpart to text chunking and advanced RAG. Reading the retrieved pages connects to document understanding.