OCR & document AI for RAG

Every RAG system rests on one assumption it rarely checks: the text it retrieves is what the document actually says. Chunking, embedding, reranking and generation all inherit whatever extraction produced. If a table cell was misread, the best retriever in the world will faithfully retrieve the wrong number.

That makes document extraction — OCR, and everything around it — the quietest high-stakes step in the pipeline.

Step zero: most pages do not need OCR

OCR (optical character recognition) means recovering text from an image of text. Many PDFs are not images. A PDF exported from Word, a reporting tool or an ERP is born digital: it contains a text layer with the exact characters and their positions. Reading that layer is fast, free and has no recognition error at all.

A scanned PDF is different: each page is a photograph wrapped in a PDF container. There is no text to read, only pixels.

So the first job is triage, per page:

• Extract the text layer with a PDF library.
• If it is present and looks like real text, use it. Done.
• If it is empty, or is garbage (some scanners add an invisible, badly recognised layer), the page is an image and needs OCR.

Three kinds of OCR

For the pages that do need it, there are three families, and a good pipeline uses more than one.

Classic engines (Tesseract, docTR) detect word boxes and recognise the text in each. They run on CPU, are fast, and return a bounding box and confidence score for every word. They know nothing about tables or reading order — a two-column page comes out interleaved.

Layout pipelines (Docling, Marker, PaddleOCR’s pipeline) first run a layout model that splits the page into regions — paragraph, table, figure, formula, header — then read each region with a specialised model and stitch the results in reading order. Much better structure; each stage can introduce its own errors.

Vision-language OCR (olmOCR, Chandra, Granite-Docling, Mistral OCR, or a general VLM prompted to transcribe) looks at the whole page image and generates Markdown or HTML, tables included. This is the best family at structure. It is also a language model, which matters a lot for the next section.

Why VLM OCR fails differently

A classic engine that cannot read a smudged digit outputs a wrong character, usually with a low confidence score and often with visibly broken text nearby. The failure is loud: a regex, a validator or a human notices.

A vision-language model generates text one token at a time, choosing likely continuations given the image and everything it has written so far. When the image is ambiguous, “likely” wins over “visible”. In a column of amounts like 1,250.00, 1,250.00, 1,250.00, a smudged fourth cell is very likely to be “read” as 1,250.00 — whether or not that is what was printed.

This is the same mechanism as hallucination in chat, appearing in a place people do not expect it. Its typical forms are completed numbers, “corrected” spellings of names and IDs, tables regularised into a neat grid with a value shifted one column, and occasional repetition loops on long dense pages.

import re

# What the VLM produced for one invoice (as parsed from its Markdown table)
vlm_lines = {"Consulting": 4812.50, "Travel": 640.00, "Software": 1199.00}
vlm_total = 6651.50

# Numeric tokens the classic engine found on the same page, with confidences
classic_words = [("4,342.50", 0.61), ("640.00", 0.97), ("1,199.00", 0.95), ("6,181.50", 0.93)]

def to_num(s):
    return float(re.sub(r"[^0-9.]", "", s))

classic_nums = {to_num(w) for w, conf in classic_words}

# Check 1: two-engine agreement on every extracted number
for item, amount in vlm_lines.items():
    status = "agree" if amount in classic_nums else "DISAGREE"
    print(f"{item:<11} {amount:>9.2f}  {status}")

# Check 2: do the line items sum to the stated total?
line_sum = round(sum(vlm_lines.values()), 2)
print(f"line items sum to {line_sum:.2f}, VLM total says {vlm_total:.2f}:",
      "ok" if line_sum == vlm_total else "MISMATCH")
print("classic engine's total on the page:", f"{to_num('6,181.50'):.2f}")

Output:

Consulting    4812.50  DISAGREE
Travel         640.00  agree
Software      1199.00  agree
line items sum to 6651.50, VLM total says 6651.50: ok
classic engine's total on the page: 6181.50

Look at what happened. The VLM misread 4,342.50 as 4,812.50 — and then wrote a total, 6,651.50, that is consistent with its own misreading. So the arithmetic check on the VLM’s output alone passes. The model did not just misread a digit; it produced a self-consistent document that disagrees with the page.

Two-engine agreement catches it, and so does comparing the VLM’s total with the total the classic engine read directly off the page. The lesson: check extracted numbers against independent evidence from the page, not only against each other.

The pipeline that holds up

Put together, a production document pipeline looks like this:

1. Triage per page. Text layer if it exists; OCR only if not.
2. Route by difficulty. Plain scanned prose to a classic engine; tables, forms, handwriting and complex layouts to VLM OCR.
3. Verify VLM output. Two-engine agreement on numbers and IDs; arithmetic where the document has internal sums; structural checks (consistent cell counts, no repeated lines, output reaching the end of the page).
4. Queue failures for humans. Only flagged pages, with the disagreement highlighted.
5. Keep provenance. Store the page number and region each chunk came from, so an answer can cite where a number was read — and a reviewer can check it in seconds.

Step 5 is what makes the finance chatbot trustworthy: it can answer “₹4,342.50, from invoice 2023-0318, page 1” and the user can click through to the image.

Measuring it on your own documents

Public benchmarks such as olmOCR-Bench score models with unit tests — small, checkable facts about each page, such as “this number appears in this row” or “this paragraph comes before that one” — rather than character error rate against a full transcript. That design is worth copying, because unit tests measure what your downstream system needs.

Build 150 to 300 of your own pages, over-sampling the hard ones, and write a few unit tests per page. Then score every route — text layer, classic engine, VLM, VLM plus checks — on accuracy, cost per page and how many errors your checks caught. When a new model is released, you can test it on your documents in an afternoon.

What to remember

• Extraction quality caps RAG quality. Nothing downstream can fix a misread number.
• Triage first: born-digital pages need no OCR.
• Classic engines fail loudly; VLM OCR fails fluently, producing plausible, self-consistent text that may not be on the page.
• Verify against independent evidence: a second engine, totals read from the page, render-and-compare.
• Keep page-level provenance so every answer can be checked.