What are chunking strategies in RAG, and how do you choose chunk size?

How to think about it

Chunking in RAG, or retrieval-augmented generation, divides source documents into smaller pieces that a retriever can search. I choose chunk size by finding the smallest unit that reliably contains the evidence needed for an answer, then validating it against retrieval quality, answer quality, latency, and cost. There is no universally correct setting such as “512 tokens.”

Why chunking changes the answer

At indexing time, an embedding model converts each chunk into an embedding, which is a numerical vector representing the chunk’s meaning. At query time, the system embeds the user’s question and finds chunks with similar vectors. The language model then receives those chunks and writes an answer from them.

That mechanism creates the central trade-off.

A very large chunk may contain the answer, but its embedding represents several subjects at once. Imagine a 1,000-token section covering pricing, cancellation, taxes, and account security. A query about cancellation may retrieve it because of the word “account,” while a more precise cancellation passage elsewhere loses the ranking contest. The model also receives more irrelevant text, which increases prompt cost and creates more opportunities to confuse an exception with the main rule.

A very small chunk has the opposite problem. A 120-token piece might contain the sentence “requests must be made within 30 days,” while the preceding sentence says what kind of request that deadline applies to. Retrieval finds the deadline, but the model does not know whether it concerns refunds, cancellations, or damaged shipments.

Overlap reduces this boundary problem by repeating some tokens between neighboring chunks. It helps when a sentence or small argument crosses a boundary, but it is not free context. Overlap increases index size, produces near-duplicate search results, and can fill the generation prompt with repeated text.

The chunk used for retrieval also does not have to be the exact text sent to the language model. A strong production design often retrieves a small, precise child chunk and then expands it to its containing section before generation.

Common misconception — chunk size is not the model’s context window. A 512-token retrieval chunk does not limit the final answer to 512 tokens. The application can combine several retrieved chunks, or return a larger parent section. The embedding model still has an input limit, though, and exceeding it may cause truncation or an error.

The main chunking strategies

Strategy	How it works	Main benefit	Main risk
Fixed-size	Split every document into windows of a chosen token length	Simple, fast, reproducible baseline	Splits sentences, sections, and tables
Sliding window	Use fixed-size chunks with repeated overlap	Preserves information near boundaries	More storage and duplicate results
Structure-aware	Split on headings, paragraphs, and sentences, falling back to smaller units when necessary	Keeps the author’s meaning and hierarchy	Requires reliable document parsing
Semantic	Group neighboring sentences and split where their meanings change sharply	Handles uneven topic boundaries	More computation and corpus-specific thresholds
Parent-child	Index small child chunks but return a larger parent section	Precise retrieval with useful context	Requires metadata and careful deduplication

These strategies can be combined. For example, I might split on headings first, split an oversized section on paragraphs, and use a small overlap only when a paragraph still exceeds the target size. Semantic splitting is useful when the source has poor headings, but it is not automatically better than a clean structural split. A complicated chunker applied to a well-structured handbook is often just a more expensive way to rediscover its headings.

A concrete example

Suppose Acme Cloud has a 24,000-token billing handbook. A user asks:

After a price increase, how long does an annual customer have to cancel and receive a refund?

The relevant section, “Billing, Price Changes, Refunds,” is about 430 tokens. It contains the 30-day rule, the definition of a price increase, and an exception for invoices already paid. Those details need to travel together.

Here is what three simple configurations look like for the whole handbook:

Chunk size	Overlap	Approximate chunks	Likely behavior
256 tokens	0	94	Precise, but the 430-token section can be split across results
512 tokens	64	54	Better boundary coverage, with moderate duplication
800 tokens	120	36	More likely to contain the whole section, but less precise

The 256-token configuration may retrieve the sentence containing “30 days” without retrieving the exception. The 800-token configuration may retrieve both, but also bring in unrelated renewal and tax material. The 512-token setting is a reasonable experiment, not a verdict.

A structure-aware splitter would first keep the heading and its paragraphs together. If the section is under the target size, it becomes one chunk. If it is too large, the splitter divides it at paragraph or sentence boundaries rather than cutting through the middle of a sentence.

A parent-child design may do even better. It could index 200-to-300-token child chunks, including the exact sentence about the deadline, while storing the 430-token section as the parent. Retrieval finds the child; generation receives the parent with the heading and exception. This separates retrieval precision from answer context.

How I choose the size

I start with the source, not with a fashionable number.

For a policy or FAQ with clear headings, I preserve sections and paragraphs. For a technical manual, I retain the heading path, such as Billing > Price changes > Refunds, in every chunk so that the embedding sees the topic even when the paragraph itself uses pronouns such as “this change.” For code, I prefer functions, classes, and related comments. For tables, I keep the header with each group of rows; a row without its column names is often meaningless.

I then examine the questions the system must answer. A question asking for one definition needs less context than a question asking for a comparison across three sections. If answers routinely require neighboring paragraphs, smaller child chunks plus parent expansion are usually safer than making every indexed chunk huge.

The embedding model’s tokenizer and input limit matter. Chunk sizes should be measured in tokens, not characters, because tokenization varies by language, punctuation, and code. A character count that works for English prose may behave badly on source code or Japanese text. I also account for the reranker and generation budget. Larger retrieved chunks consume more prompt tokens even when the final answer is short.

For ordinary prose, I might test 256, 512, and 768 tokens with zero, 10 percent, and 20 percent overlap. That is an initial grid, not a rule. I keep the embedding model, retriever, reranker, prompt, and top result count fixed while comparing configurations.

The evaluation set should contain representative questions paired with the source passage that supports each answer. Even 50 carefully chosen questions are more useful than intuition alone. I measure retrieval recall at k, meaning whether a supporting chunk appears among the first k results, along with final answer correctness and citation accuracy. I also record index size, prompt-token usage, and p95 latency. A setting that raises recall but doubles cost may be the wrong production choice.

Failure modes I look for

The first symptom of a boundary problem is often an answer such as “the policy does not specify that,” even though a manual search finds the sentence immediately. I inspect the retrieved chunks. If the condition is in one chunk and the deadline or exception is in the next, I try structure-aware splitting, modest overlap, or parent expansion.

If the top five results are nearly identical, overlap may be too large, or the retriever may be returning several windows from the same passage. I deduplicate by source section and preserve a few genuinely different results.

Another common failure is a confident answer that cites the main rule but omits an exception. Larger chunks can help, but retrieving the whole parent section is usually a cleaner fix than doubling every chunk in the index.

I would not use semantic chunking just because it sounds sophisticated. It adds computation and a similarity threshold that may work for one document collection and fail for another. If headings and paragraphs already match how people ask questions, structure-aware chunking is easier to debug and maintain.

What they’ll ask next

“Why not use the largest chunk the embedding model allows?”

Because the maximum input size is a ceiling, not an ideal. Large chunks mix topics, dilute similarity, increase prompt cost, and may hide a precise answer among irrelevant passages. I would test larger chunks only when answers genuinely require broad context.

“Do you always need overlap?”

No. Good structural boundaries, parent-child retrieval, or neighboring-section expansion can make overlap unnecessary. I use overlap when important evidence commonly crosses boundaries, then check whether it improves recall enough to justify duplicate storage and retrieval results.

“How do you handle tables, PDFs, or code?”

I preserve the source’s logical units before choosing a token size. A table chunk should include its column headers and relevant rows. A code chunk should normally follow a function or class boundary. A PDF may need layout-aware extraction first; no chunking strategy can repair text that was extracted in the wrong reading order.

Say this in the interview

“Chunk size is a retrieval and context-budget decision, not a magic number: I preserve document structure, start with measured token and overlap baselines, and choose the smallest unit that reliably contains the evidence, validated on recall, answer quality, latency, and cost.”