Your RAG system retrieves the right document but the answer misses a qualification contained in a table, code block, or heading several paragraphs away. How would you redesign chunking and parent-child retrieval without causing the prompt to grow uncontrollably?

How to think about it

I would make retrieval units small and structure-aware, then use their parent sections only as bounded evidence expansion—not as permission to paste an entire document into the prompt. Each child hit would carry its section path and links to nearby headings, tables, code blocks, and explicit references; a reranker and token budget would select only the context needed to include both the answer and its qualification.

Why the failure happens

Retrieving the right document is not the same as retrieving the right evidence.

A typical RAG system, or retrieval-augmented generation system, embeds text into vectors, finds chunks near the user’s question, and places those chunks in the model prompt. Vector similarity is useful, but it is not a document-understanding engine. A paragraph saying “use refresh tokens to maintain a session” may be highly similar to the question. A table row saying “browser clients: rotation required” may use different words and rank lower.

The model cannot reliably use evidence it never sees. It also cannot infer that a table four paragraphs later qualifies an earlier sentence merely because both came from the same document.

There is a second trap: headings and formatting carry meaning. “Caching” under “Development defaults” means something different from “Caching” under “Production restrictions.” If a chunk contains only the paragraph and loses its heading path, the model has lost the scope of the rule.

The useful distinction is:

• Document recall: did retrieval find the correct document?
• Evidence recall: did it find the passages, rows, or code that support the complete answer?
• Qualification recall: did it find the exception, limit, or condition that prevents an incomplete answer?

The interview scenario says document recall is already working. The redesign should target the latter two.

The chunking design

I would parse the source into a structural tree before making chunks:

document → heading → subsection → paragraph, list, table, code block

A prose chunk might contain 200 to 400 tokens, but that is a starting range, not a law. A sentence that defines a rule and its exception should stay together even if it produces a 430-token chunk. A paragraph boundary is usually more meaningful than cutting exactly at token 300.

Every child chunk should retain compact structural metadata:

• document and version
• parent section identifier
• full heading path
• block type: paragraph, list, table, code, or heading
• position in the parent
• links to referenced blocks such as “see Table 2”

The heading path should be available to retrieval and reranking, not merely stored in a database field that the model never sees. A compact prefix such as Authentication / Refresh tokens / Browser clients gives both the embedding model and the reranker useful scope.

Tables deserve special handling. Keep the table title, column headers, row labels, and footnotes together whenever possible. If a large table must be split, repeat the title and headers in every child. A row containing “required” is nearly useless without knowing which column contains that word.

Code blocks should normally remain atomic. Preserve the language, the immediately preceding explanation, and any comment that states a constraint. Splitting a configuration block between a setting and its value is a particularly efficient way to retrieve nonsense.

The child is the search unit. The parent is the context unit.

Parent-child retrieval without prompt bloat

At query time, I would use a staged process.

First, retrieve a modest number of child candidates, using hybrid search where appropriate. Vector search handles paraphrases; lexical search is valuable for exact setting names, table labels, error messages, and version numbers. Then rerank those candidates against the question.

Next, group the winning children by parent section. Do not blindly include every child from every parent. For each parent, expand only through structural relationships:

1. the matched child;
2. its heading path;
3. a nearby definition or prerequisite;
4. a linked table, code block, footnote, or exception block when relevant.

Finally, pack the evidence under explicit limits. For example, I might allow at most two parent sections, three evidence blocks per section, 700 tokens per parent, and 1,800 evidence tokens overall. Those are operational starting points, not universal constants. The important property is that the limits are global and deterministic.

The packer should deduplicate text. If the child already contains a table row, it should not also paste the same row inside a full table. It should include the table title, headers, relevant row, and footnote instead.

A generated summary of the parent can help choose which blocks to expand, but I would not use that summary as the only evidence. Summaries are excellent at making a 1,100-token section look tidy and occasionally excellent at deleting the one sentence that matters.

A concrete example

Suppose a nine-page internal deployment guide has this section:

Authentication → Refresh tokens → Client policies

The section contains a 280-token explanatory paragraph, followed four paragraphs later by a table and then a 75-token configuration example.

Client type	Storage	Rotation
Browser SPA	Memory only	Required
Server application	Encrypted server store	Recommended

The user asks, “How should a browser application handle refresh tokens?”

A conventional fixed-size splitter may retrieve the explanatory paragraph. It may correctly identify the document while missing the table row. The resulting answer says, “Use refresh tokens to renew sessions,” which sounds reasonable and omits the important restriction.

During ingestion, I would represent the row with its headers and section scope:

{
  "text": "Client type: Browser SPA; storage: memory only; rotation: required",
  "block_type": "table_row",
  "table_title": "Client policies",
  "section_path": ["Authentication", "Refresh tokens", "Client policies"],
  "parent_id": "auth/refresh-tokens/client-policies"
}

The paragraph may win initial retrieval. Its parent relationship then points to the policy table. The context packer adds the heading path, the paragraph, the browser row, and the relevant footnote, perhaps 550 tokens altogether. It does not add all nine pages or even the entire 1,100-token parent section.

The model now has enough context to answer: browser applications use memory-only storage, and rotation is required. The qualification survived because the retrieval system represented the document’s structure instead of treating it as a bag of equally shaped text fragments.

The senior-level trade-off

Small chunks improve precision but reduce context. Large chunks preserve context but introduce irrelevant material, consume tokens, and can bury the useful sentence among contradictory defaults. Parent-child retrieval is a compromise: search narrowly, expand selectively.

It is not magic. A qualification that is genuinely remote and has no structural or textual link can still be missed. For high-risk material such as security, pricing, or compliance policies, I would add explicit relationships during ingestion: rule to exception, claim to footnote, setting to valid-value table. I would also keep document versions attached, because expanding a current paragraph with a stale table is worse than retrieving nothing.

The first production symptom of a bad design is often not a retrieval error. It is an answer that is broadly correct but confidently missing the condition: “rotation is recommended” instead of “rotation is required for browser clients.” Another symptom of over-expansion is a vague answer with several caveats from unrelated sections, rising latency, and prompts that approach the model’s context limit.

I would evaluate this with a test set containing deliberately separated qualifications. Measure evidence recall, qualification recall, citation support, answer correctness, and prompt tokens per query. A system that retrieves the right document on every test but misses the exception on one in five questions is not ready. The exception is usually the whole point.

What they’ll ask next

How would you choose child and parent sizes?
Start with structure rather than a universal token number. Test prose chunks around 200 to 400 tokens, keep tables and code intact, and measure qualification recall against a fixed token budget. Increase size only when the evaluation shows that definitions or exceptions are being separated.

How would you handle a table with 500 rows?
Index the table title, headers, row groups, and individual rows with repeated headers. Retrieve the relevant rows, but include the title, headers, and any applicable footnote. A row without its column names is not evidence; it is a small, well-formatted ambiguity.

What prevents a query with many hits from filling the prompt?
Use a global token budget, per-parent limits, deduplication, and score-based selection. If the budget is exceeded, keep the blocks with the highest marginal value rather than truncating the end of the prompt. Log which evidence blocks were dropped so failures can be diagnosed.

Say in the room: “I would retrieve small, structure-aware children, then expand only the winning parents and their linked qualifications under a hard evidence budget.”