OCR that actually works in 2026: stop picking a model, start building a router
Vision-language models finally read tables, handwriting and multi-column layouts well enough to ship. They also fail in a new way: fluent, confident text that is not on the page. The teams that get document AI right route each page to the cheapest engine that can read it, and verify the expensive one.
Ask a team that runs document AI in production what their OCR accuracy is and you will get a number like 98 or 99 percent. Ask them why the project is late and you will hear about a table.
That gap is the whole story of OCR in 2026. Reading characters is a solved problem for clean print, and has been for years. What still breaks pipelines is everything around the characters:
- Structure. Which cells belong to which row. Which column a number sits under once a header spans two of them.
- Reading order. A two-column page read straight across produces sentences that interleave two unrelated paragraphs.
- Forms. A ticked checkbox, a handwritten amount, a stamp over a signature line.
- Silent errors. A wrong digit in an invoice total that looks exactly like a right one.
The last one is the subject of most of this post, because the newest and best OCR models have made it more likely, not less.
Three generations, all still in use
It helps to see the field as three generations stacked on top of each other, because a good pipeline uses all three.
1. Classic engines. Tesseract, and the deep-learning successors of that design such as docTR. A text detector finds word boxes; a recognizer reads each box. Fast, cheap, runs on a CPU, and gives you a bounding box and a confidence for every word. They know nothing about tables or reading order.
2. Layout pipelines. Docling, Marker, PaddleOCR’s pipeline mode, Unstructured. A layout model first segments the page into regions — paragraph, table, figure, header, formula — then each region goes to a specialised reader, and a final step stitches them together in reading order. Much better structure; each stage is a place where errors can compound.
3. End-to-end vision-language OCR. A single VLM looks at the page image and writes out the whole thing as Markdown or HTML, tables and equations included. olmOCR 2 from Ai2, Datalab’s Chandra, IBM’s Granite-Docling, Mistral OCR, and general frontier VLMs used with an OCR prompt all sit here. Best at structure, slowest per page, and the generation with the new failure mode.
How good the VLM generation actually is
The most useful public yardstick right now is olmOCR-Bench, released by Ai2 alongside olmOCR. Instead of scoring edit distance against a reference transcript, it runs thousands of small unit tests against each page’s output: this sentence is present, this number appears in the cell under this header, this paragraph comes before that one, this header text does not leak into the body. That matches how a downstream system actually consumes OCR, which is why it has become the benchmark vendors quote.
Overall scores as reported in Datalab’s Chandra repository (most of these are vendor-run, so treat small gaps as noise):
| System | olmOCR-Bench overall |
|---|---|
| Chandra 2 (Datalab, open weights) | 85.8 |
| dots.ocr 1.5 | 83.9 |
| olmOCR 2 (Ai2, open weights) | 82.4 |
| Marker 1.10 (pipeline) | 76.5 |
| DeepSeek-OCR | 75.4 |
| Mistral OCR (API, 2025 version) | 72.0 |
| GPT-4o, with text-layer anchoring | 69.9 |
| Qwen3-VL 8B, general-purpose | 64.6 |
Two things stand out.
First, specialised beats general. Small models trained specifically for OCR beat much larger general-purpose models that are merely prompted to transcribe. Reading a dense page faithfully is a narrow skill, and it rewards narrow training.
Second, look at how olmOCR 2 got there. Ai2 fine-tuned it with reinforcement learning where the reward was the fraction of synthetic unit tests a page’s output passed — the same idea that changed agent training: if you can check an answer automatically, you can train against the check.
The new failure mode: fluent fabrication
When Tesseract cannot read a smudged digit, it emits a wrong character, often with a low confidence score, and frequently produces visibly broken text around it. Ugly — but loud. A reviewer or a regex notices.
A VLM is a language model. When it cannot read a smudged digit, it does what language models do: it produces the most plausible continuation. In a table of monthly figures, the most plausible value for an unreadable cell is one that looks like its neighbours. The output is clean, well-formatted, and wrong, and nothing about it looks wrong.
The same mechanism produces a family of failures you will meet in production:
- Completed numbers. A partly visible amount is “read” in full.
- Repaired text. A typo or an unusual spelling on the page is silently corrected to the common form — fatal for names, IDs and drug names.
- Regularised tables. A ragged table with a merged cell comes back as a tidy grid, with one value shifted a column.
- Repetition loops. Long dense pages occasionally make the model repeat a row or a line until it hits its token limit.
None of this means VLM OCR is unusable. It means its output needs a different kind of check than classic OCR did.
The architecture that works: route, then verify
The single most effective cost and quality decision in document AI is not running OCR on pages that do not need it. A large share of enterprise PDFs are born digital — exported from Word, an ERP or a reporting tool — and already carry an exact text layer. Extracting it with a PDF library is effectively free and has zero recognition error. OCR is for pages that are images.
Step 1: triage every page
Triage is cheap and decides most of your bill:
- Does the page have a usable text layer? Extract it and check it is real text rather than an invisible, garbled layer some scanners add. If it is clean, you are done — no OCR.
- Is it a scan of plain prose? A classic engine on CPU handles it, and gives you word-level confidences for free.
- Does it contain tables, forms, formulas, handwriting or a complex multi-column layout? That is what the VLM is for. A small layout detector — or the classic engine’s own block output — is enough to decide.
Mixed documents are routed per page, not per file. A 60-page annual report might be 50 pages of born-digital prose and 10 pages of scanned financial tables.
Step 2: verify what the VLM says
Every check below is cheap compared with the VLM call it guards:
Two-engine agreement. Run the classic engine on the same page and
compare the numbers and identifiers both produced. Classic OCR is
weak at structure but rarely invents a digit with high confidence. When
the VLM says 4,812.50 and the word boxes in that region say
4,342.50, you have found exactly the error you were worried about.
Arithmetic. Financial documents are full of internal checksums: line items sum to a subtotal, tax is a known rate of it, columns total to a stated total. Recompute them. A failed sum flags a misread far more reliably than any confidence score.
Render and compare. Render the VLM’s Markdown table back to an image, or align its cells against the word boxes from step one, and measure how much printed text on the page has no counterpart in the output — or the reverse. Text that exists in the output but nowhere on the page is a fabrication by definition.
Structural sanity. Rows with a different number of cells from the header, a line repeated many times, or output that stops before the page’s last visible text all signal a decoding failure. Retry once at a different resolution before sending the page to a human.
It is 0.08 × 0.06, roughly half a percent of pages. That is the point of the design: reviewers look at a small, pre-flagged set of pages rather than sampling everything and hoping.
Choosing models — including where they come from
Many teams now have a procurement rule about model origin, so it is worth knowing provenance along with accuracy. Provenance is also more tangled than it looks. olmOCR 2 is published by a US non-profit under Apache 2.0, and is a fine-tune of Alibaba’s Qwen2.5-VL-7B. A rule written as “no Chinese models” has to decide what to do with that.
| Option | Who | Type | Licence and deployment notes |
|---|---|---|---|
| Text-layer extraction | — | Library | Not OCR at all; use it first |
| Tesseract | Open source | Classic engine | Apache 2.0, CPU, fully local |
| docTR | Mindee (France) | Classic engine | Apache 2.0, local |
| Docling + Granite-Docling-258M | IBM | Pipeline + small VLM | Apache 2.0, local, runs on modest hardware |
| Marker / Surya | Datalab (US) | Pipeline | Check the licence for commercial use |
| Chandra 2 | Datalab (US) | VLM OCR | Code Apache 2.0; weights under a modified OpenRAIL-M licence — free below a revenue/funding threshold, commercial licence above it |
| olmOCR 2 | Ai2 (US) | VLM OCR | Apache 2.0; fine-tuned from Qwen2.5-VL |
| Mistral OCR | Mistral AI (France) | Hosted API | Pages leave your network unless you arrange a self-hosted deployment |
| PaddleOCR / PaddleOCR-VL | Baidu | Pipeline + VLM | Apache 2.0, local |
| DeepSeek-OCR | DeepSeek | VLM OCR | Open weights, local |
Whatever your origin policy, the decision that actually matters for data protection is where inference runs. Open weights running on your own hardware send nothing anywhere, whoever trained them. A hosted API sends every page to someone else’s servers, whoever trained that. We unpack this properly in a threat model for Chinese open-weight models.
Build your own benchmark before you build anything else
The single best investment in a document AI project is a small golden set, built before choosing a model:
- Sample 150–300 real pages, stratified by document type and by difficulty. Over-sample the ugly ones: faxes, phone photos, stamps, rotated pages, handwriting.
- Write unit tests, not transcripts. Copy olmOCR-Bench’s idea: “the invoice total is 4,812.50”, “row ‘Freight’ has value 120.00 in column ‘Amount’”, “the footer text does not appear in the body”. They take minutes per page to write and directly measure what your downstream system needs.
- Score every candidate route, not just every model: text layer alone, classic engine, VLM, VLM plus verification. Record cost per page and pages per second alongside accuracy.
- Measure the verifier, too. Of the VLM errors on your set, what fraction did your checks catch? That number, not the raw accuracy, is what tells you how many silent errors reach production.
This also protects you from the next leaderboard. When a new model lands with a higher score, you can run it against your 200 pages in an afternoon and know whether it matters for your documents.
What to take away
- OCR is no longer about characters. It is about structure and about catching silent errors.
- Specialised VLM OCR models are now genuinely good at structure, and beat much larger general models prompted to transcribe.
- They fail by fabricating plausible text. Verification with a second engine, arithmetic and render-and-compare is not optional.
- Route per page: text layer first, classic engine for simple scans, VLM only where it earns its cost.
- Provenance policy and data protection are different questions. Where inference runs decides the second one.
For the parsing side of the same problem — turning extracted pages into chunks a retriever can use — see PDF parsing remains unsolved and the 50,000-document pipeline.