Chunking for RAG

RAG retrieves “chunks” — but a document isn’t born as chunks. Before anything can be embedded and indexed, you have to split documents into pieces, and how you split is one of the highest-leverage, most underrated decisions in a RAG system. Get it wrong and even a great embedding model retrieves the wrong thing.

The chunk-size trade-off

Everything hinges on one tension:

• Chunks too big — the one relevant sentence is diluted by surrounding irrelevant text, so its embedding is a muddy average and retrieval precision drops (and you waste context-window space feeding the model padding).
• Chunks too small — a chunk loses the context needed to understand or answer (a pronoun whose antecedent is in the previous sentence, a number whose units are in the heading), so even a correct retrieval is useless.

The basic mechanics are a window size and an overlap that carries context across boundaries:

doc = "RAG retrieves relevant chunks then the LLM generates a grounded answer from them".split()

def chunk(words, size, overlap):
    step = size - overlap
    return [" ".join(words[i:i+size]) for i in range(0, len(words), step) if words[i:i+size]]

for size, overlap in [(4, 0), (6, 2)]:
    cs = chunk(doc, size, overlap)
    print(f"size={size} overlap={overlap}: {len(cs)} chunks")
    for c in cs:
        print(f"  | {c}")

size=4 overlap=0: 4 chunks
  | RAG retrieves relevant chunks
  | then the LLM generates
  | a grounded answer from
  | them

size=6 overlap=2: 4 chunks
  | RAG retrieves relevant chunks then the
  | then the LLM generates a grounded
  | a grounded answer from them
  | them

With no overlap, “generates a grounded answer” is split across two chunks — neither alone says what’s generated. The overlap=2 version repeats “then the” / “a grounded” across boundaries, so a sentence straddling a cut still lands intact in some chunk. Overlap is cheap insurance against splitting a thought in half.

Strategies, from naive to smart

• Fixed-size — split every N tokens. Simple and fast, but blindly cuts mid-sentence; almost always paired with overlap.
• Sentence / paragraph-aware — split on natural boundaries so chunks are coherent units.
• Structure-aware — respect the document’s own structure: Markdown headings, sections, code blocks, table rows. Keeps semantically-whole units together.
• Semantic chunking — split where the topic shifts (detect drops in sentence-to-sentence similarity), so each chunk is about one thing.
• Parent-document retrieval — index small chunks for precise matching, but return the larger parent (section/page) to the LLM, getting precision and context.

In one breath

• Chunking splits documents before embedding/indexing, and how you split is a top RAG-quality lever — a great embedder can’t fix bad chunks.
• The size trade-off: too big dilutes the relevant sentence (muddy embedding, wasted context); too small strips the context a chunk needs to be understood.
• Overlap carries context across boundaries so a thought split by a cut still lands whole in some chunk (the demo: overlap repeats then the across the seam).
• Strategies: fixed-size + overlap, sentence/paragraph, structure-aware (headings/code), semantic (split on topic shift), and parent-document (index small, return the parent).
• There’s no universal best — match chunking to your documents and queries, attach metadata, and measure it with RAG evals rather than guessing.

Quick check

Next

Chunks become vectors via sentence embeddings stored in a vector database; the retrieval pipeline is RAG basics and advanced RAG, and you tune chunking by measuring with RAG evals.