Chunking for RAG

At 3:07 a.m., someone asks your internal support bot:

Can a contractor approve a refund above $500?

The answer is in the company handbook. It is one sentence under a heading that says Contractor purchases. Your system retrieved the sentence, but not the heading. The model confidently answers, “Yes, managers can approve refunds above $500,” because a nearby paragraph described managers. The retrieval score looked healthy. The answer is still wrong.

This is the problem chunking exists to solve.

A document arrives as a page, PDF, Markdown file, transcript, or database record. Before a RAG system can embed and index it, you split it into chunks: pieces of text that will be matched against a user’s question. A good chunk is small enough to match precisely and large enough to carry the meaning needed for an answer.

That sounds like housekeeping. It is not. If the relevant fact is mixed with noise, or its qualification is cut away, a better embedding model cannot reconstruct what you threw out.

What a chunk does to retrieval

A typical RAG retriever does not compare a question with an entire 40-page handbook. It compares the question with indexed chunks.

First, an embedding turns each chunk into a vector representing patterns in its meaning. The question gets its own vector. The retriever ranks chunks whose vectors are most similar to the question vector, and the top results become context for the LLM.

That gives each chunk two jobs:

1. It must contain enough signal to match the question.
2. It must contain enough context to support the answer.

Large chunks often fail the first job. Small chunks often fail the second.

Imagine a deliberately simple embedding with two dimensions: “refund policy” and “general billing.” A 200-token chunk contains 40 tokens about refund approval and 160 about unrelated billing procedures. In this toy arithmetic, refund material makes up 20 percent of the chunk. An 800-token chunk containing the same 40 relevant tokens has only 5 percent refund material.

Real embeddings are not literal word-count averages. But the intuition holds: mixing unrelated ideas makes a chunk’s representation less specific. A refund query may now compete with queries about invoices, payment dates, and billing addresses. The noisy chunk is harder to retrieve and, once retrieved, wastes context-window space.

At the other extreme, a 35-token chunk might say:

Refunds above $500 require director approval.

If the heading and previous sentence say that this applies only to contractor purchases, the chunk matches the query while omitting its scope. It is locally precise but globally misleading. Chunking determines the unit of meaning that retrieval can see.

Size and overlap, with actual numbers

A fixed-size chunker chooses a window of size tokens and moves forward by size - overlap tokens. Overlap is the repeated text at the edge of consecutive chunks.

doc = "RAG retrieves relevant chunks then the LLM generates a grounded answer from them".split()

def chunk(words, size, overlap):
    step = size - overlap
    return [" ".join(words[i:i+size]) for i in range(0, len(words), step) if words[i:i+size]]

for size, overlap in [(4, 0), (6, 2)]:
    cs = chunk(doc, size, overlap)
    print(f"size={size} overlap={overlap}: {len(cs)} chunks")
    for c in cs:
        print(f"  | {c}")

Overlap helps when a sentence or idea crosses a boundary, but it increases embedding work and can produce duplicate results. A naïve implementation can also create a useless tiny tail; production chunkers usually merge that tail into the previous chunk or apply a minimum-size rule.

Tokens are not words. A tokenizer may represent a common word as one token and a technical identifier as several. Choose sizes in the units your embedding model receives, and check its input limit: an API may truncate an oversized chunk or reject it.

Here is the arithmetic for a 1,000-token section, chunks of 240 tokens, and 40 tokens of overlap. The step is 200 tokens, so chunks begin at positions 0, 200, 400, 600, and 800. That produces five chunks:

• four full chunks of 240 tokens: 960 tokens
• one final chunk of 200 tokens: 200 tokens
• total processed text: 1,160 tokens

The original had 1,000 tokens. Overlap caused 160 token positions to be processed again, a 16 percent increase. Those repeated tokens cost embedding work and storage. They can also cause the retriever to return several nearly identical chunks for one fact, leaving less room for other evidence.

Overlap is insurance, not magic.

A useful starting point is modest overlap, perhaps 10 to 20 percent of the chunk size, but test it against real questions. Self-contained FAQ answers may need none; policies with qualifications split across sentences may benefit from it.

Strategies, from simple to structure-aware

Fixed-size chunking splits every document into the same token window. It is fast, deterministic, and an excellent baseline, but it knows nothing about sentences, headings, code, or tables.

Sentence- or paragraph-aware chunking adds text until the next sentence or paragraph would exceed the target size. It avoids many mid-sentence cuts, though a paragraph may still contain unrelated ideas. Keep a maximum and a fallback for unusually large paragraphs.

Structure-aware chunking follows the document hierarchy. Keep a Markdown heading with its section, code blocks intact, table headers with rows, and page or subsection identifiers as metadata. This is usually the strongest first choice for documentation, policies, API references, and well-formed Markdown.

Preserve structure in the chunk text, not only in metadata. Contractor purchases > Refunds gives both the embedder and answer model more scope than a chunk beginning with Refunds.

Semantic chunking detects topic changes, often by embedding consecutive sentences and splitting when similarity drops sharply. It can help badly formatted prose, but adds ingestion work and requires corpus-specific thresholds. A topic shift in a policy may be a crucial qualification rather than a safe split.

Parent-document retrieval separates matching size from answer size. Index a small child, such as a paragraph, for precise matching. When it wins, return a bounded parent, such as the complete subsection or page, to the LLM. The child finds the needle; the parent supplies the surrounding thread.

Do not make the parent a 6,000-token chapter and recreate whole-document retrieval. Keep it bounded, retain the matching child as an anchor, and deduplicate when several children point to the same parent.

The right granularity follows the structure of the corpus and the questions. “What status code means throttled?” may need an error block; “What is our incident response process?” may need a whole section.

A production pattern

A reliable ingestion pipeline does more than call split(text, 500).

Parse sources while preserving titles, heading paths, code fences, lists, table boundaries, pages, and offsets. Normalize repeated whitespace without erasing labels that explain scope. Start with structure-aware splitting for clean documentation, or paragraph-aware splitting with a maximum for messy text. Add overlap only where a boundary can plausibly divide a thought.

Attach metadata to every child chunk: document ID, version, source URL, page or section, heading path, access-control scope, and character or token offsets. This supports filtering, citations, stale-version cleanup, and debugging. For parent retrieval, store an explicit parent ID; retrieve children, optionally rerank them, group hits by parent, and add only the best few bounded parents to context.

Evaluate chunking with real questions whose correct source sections are labeled. Measure retrieval recall at k, or how often the correct source appears among the first k results. Inspect the final answer too: was the relevant qualification present, and was the right version cited?

Suppose 100 questions produce recall at 5 of 68 percent with fixed 500-token chunks. A structure-aware version reaches 84 percent, but answer accuracy rises only from 72 to 75 percent. Retrieval improved, yet parent context may still be too large, or reranking and answer constraints may be the remaining problems. RAG evals help separate those causes.

Failure modes you can see in production

A high-scoring result produces a confidently incomplete answer. Inspect the chunk, not just its similarity score. If it contains the rule but not its heading, exception, unit, or definition, change the boundary, repeat the section path, or retrieve a bounded parent.

The context window fills with near-duplicates. Excessive overlap or parent expansion may make several hits point to the same text. Count unique parents, reduce overlap, and deduplicate before generation.

Table questions produce fluent nonsense. Values were probably separated from column names or units. Keep headers with rows and test questions that require comparing rows.

Code answers are broken. Do not split code by ordinary sentence rules. Keep functions, request examples, and response objects intact where possible; for large files, split at functions or classes and retain file and symbol metadata.

Also track versions. Delete or deactivate old children atomically so stale parents and current children cannot be retrieved together.

Chunking is worth avoiding when source units are already self-contained: an 80-token FAQ answer, a database row with its schema, or a short product specification may worsen when split. Semantic chunking is also a poor first move when deterministic rebuilds, simple debugging, or ingestion cost matter more than gains on messy text.

Long-context models change the economics, not the logic. A larger parent may fit, but it still costs tokens, can bury evidence among distractions, and cannot restore text absent from retrieval.

In one breath

• A chunk is the unit your retriever embeds and ranks. Its boundaries determine both what can match a question and what context reaches the LLM.
• Large chunks dilute retrieval signals; tiny chunks can omit headings, definitions, units, or exceptions.
• Overlap protects against ordinary boundary cuts, but increases index work and cannot restore missing context.
• Start with structure-aware or paragraph-aware splitting. Use semantic chunking when messy sources justify its extra ingestion work.
• Parent-document retrieval uses small children for matching and bounded parents for context. Deduplicate parents.
• Preserve structure, metadata, versions, and access scope. Measure retrieval recall and final answer quality on real questions.

Quick check

Next

Chunks become vectors via sentence embeddings stored in a vector database. The full retrieval pipeline is covered in RAG basics and advanced RAG, while RAG evals show how to tune chunking without guessing.