LlamaIndex: indexes, query engines & retrievers

It is 3 a.m. A support agent asks your internal assistant:

“Can a Starter customer get a refund after cancelling on day 31?”

The company handbook is in a PDF. The model has never been trained on that handbook. You add the PDF to the prompt anyway. The answer comes back:

“Yes. Customers can request a full refund within 60 days.”

The document says 30 days. The model sounded certain because language models produce plausible sentences. They are not automatically connected to your files.

A retrieval-augmented generation system, or RAG, fixes the missing connection by doing three things:

• finding relevant pieces of your data;
• placing them in the model’s context; and
• asking the model to answer from them.

The difficult part is turning PDFs, pages, tickets, and database rows into evidence that can be found reliably.

LlamaIndex is a Python and TypeScript framework for that data side of an LLM application. It names the replaceable stages: loading, chunking, indexing, retrieving, and composing the response.

The LlamaIndex vocabulary

LlamaIndex’s basic pipeline has five nouns. They describe preparing knowledge and answering questions.

• A Document is content loaded from a PDF, web page, file, or database record.
• A Node is a smaller piece of a Document, carrying text and metadata such as a filename, page number, customer ID, or document date.
• An Index is a structure built over Nodes so relevant ones can be found. VectorStoreIndex is common for semantic search: it stores an embedding, a list of numbers representing a Node’s meaning.
• A Retriever receives a query and returns the Nodes that appear most relevant.
• A Response Synthesizer turns the query and retrieved Nodes into an answer, usually by calling an LLM.

A Query Engine conveniently wraps a Retriever and a Response Synthesizer. Calling query_engine.query(...) hides those steps, but they still happen.

The split looks like this:

The important boundary is offline preparation versus online answering.

For the common VectorStoreIndex path, preparation loads documents, splits them into Nodes, creates embeddings, and writes an index. During a question, the system embeds the query, searches the index, selects a small set of Nodes, and gives them to the synthesizer.

Keyword, full-text, and metadata-filtered paths use different transformations, but the separation remains.

“Offline, once” means once per version of the data. When sources change, the affected parts must be indexed again.

What each stage actually does

Documents become Nodes

A 40-page handbook may contain vacation policy, expense rules, and contractor exceptions. Returning the whole PDF for every question wastes context and makes the relevant sentence harder to find.

A node parser splits a Document into Nodes according to character or token counts, sentences, paragraphs, headings, or document structure. A Node contains:

• text to embed and retrieve;
• inherited metadata;
• an identifier for tracing it to the source.

Chunk size is a trade-off. Tiny Nodes are precise but can lose the condition that gives a sentence meaning. Huge Nodes preserve context but dilute similarity and consume more context.

A small overlap can keep an exception connected to its rule, but excessive overlap duplicates evidence and enlarges the index.

Nodes become an Index

An embedding model converts each Node into a vector: a point in a high-dimensional space. Nodes about refunds should be closer to questions about getting money back than to questions about resetting a password, even when the wording differs.

A vector index stores those vectors and supports nearest-neighbour search. The query is embedded with the same model, and the index returns nearby Nodes, commonly using cosine similarity or a related distance.

The vector is a search aid, not a compressed copy of the truth. It may miss exact identifiers, invoice numbers, product codes, or negations.

Keyword search and metadata filters often handle those signals better, which is why production systems frequently combine retrieval methods.

A LlamaIndex VectorStoreIndex is the logical object that works with vectors. A vector store is the storage system holding them. It may be local during a prototype or backed by a persistent database in production.

A Retriever selects evidence

A retriever takes a question and returns ranked Nodes. Top-k is the number of results; in LlamaIndex vector APIs it is commonly configured with similarity_top_k.

A retriever may use:

• vector similarity;
• keyword or full-text search;
• metadata filters;
• hybrid scoring; or
• a reranker.

Retrieval quality is the foundation of answer quality: a fluent response built from the wrong Nodes is still wrong. The model cannot quote evidence it never received.

The synthesizer writes the answer

The response synthesizer receives the query and retrieved context, formats them into a prompt, asks the configured LLM to answer, and returns a response object.

It may produce a compact answer, summary, or multi-chunk synthesis depending on its response mode.

The synthesizer is not a fact checker. It can merge incompatible policies or answer from general knowledge when the context is silent. Instructions to say “I don’t know” when context is insufficient help, but evaluation and application checks remain necessary.

source_nodes exposes the evidence used by the response. Use it for citations, debugging, and evaluation traces. It does not make an unsupported answer true.

The default path

Sensible defaults connect the stages:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What's our refund window?")
print(response)
print(response.source_nodes)

SimpleDirectoryReader loads files as Documents. VectorStoreIndex.from_documents parses, embeds, and indexes them. as_query_engine() creates the Retriever and Response Synthesizer; the final call retrieves context and synthesizes an answer.

This is a prototype, not a production design. You still need:

• persistence;
• update handling;
• authorization;
• evaluation; and
• a response for cases where retrieval finds nothing.

The components are replaceable: change the parser, vector store, retrieval strategy, or reranker without changing the overall mental model.

Follow one question with numbers

Suppose a refund handbook contains about 6,000 words, or roughly 8,000 tokens after tokenisation. You choose 500-token chunks with a 50-token overlap.

The first Node covers tokens 1 through 500. The next starts at token 451, so each additional Node contributes 450 new tokens. The approximate number of Nodes is:

ceil((8,000 - 50) / (500 - 50)) = ceil(7,950 / 450) = 18

Those 18 Nodes are embedded during indexing. A customer asks, “Can I get money back after cancelling on day 31?” The query is embedded once. The retriever might return:

1. Starter refund terms, similarity score 0.86.
2. Enterprise refund terms, score 0.79.
3. Cancellation procedure, score 0.63.

The response context is about 1,500 tokens before the question and system instructions. Returning eight Nodes would provide about 4,000 tokens.

That could recover a missed exception, but it costs more context and gives the synthesizer more chances to combine unrelated policies.

For 1,000 questions, a prebuilt index creates 18 document embeddings once and 1,000 query embeddings. Scanning and embedding all 18 Nodes afresh for every question would create about 18,000 document embeddings.

The index amortises repeated work.

If Starter refunds change from 30 days to 14, the old Node remains wrong until the index is updated. A robust ingestion process identifies changed documents, replaces their Nodes, and removes deleted content. “Offline” never means “ignore updates.”

Choosing retrieval in practice

There is no universally best retriever. Choose based on what makes a match relevant.

If a user from tenant A asks about “our refund policy,” retrieving tenant B’s policy and trusting the model to ignore it is not access control. Apply permissions before or during retrieval.

A reranker helps when first-stage retrieval finds plausible candidates but orders them poorly. It adds latency and usually another model call, so use it where measured retrieval errors justify the cost.

Failure modes you can actually diagnose

Fluent answer, wrong policy

Inspect source_nodes before changing the prompt. Log the query, Node IDs, metadata, scores, and final context.

If the right passage is absent, improve parsing, chunking, filters, retrieval method, or similarity_top_k. If it is present but ignored, investigate context ordering, conflicting passages, and the synthesizer prompt. The model cannot retrieve a missing paragraph by “trying harder.”

A PDF produces nonsense Nodes

Repeated headers, interleaved table columns, and misplaced page numbers indicate an ingestion problem.

Use a loader or parser suited to the document, preserve page and section metadata, and inspect representative Nodes before embedding them. For difficult PDFs, LlamaParse may be more appropriate than plain text extraction.

New policy text never appears

Use stable source IDs and version or effective-date metadata. Make ingestion idempotent, so rerunning it creates one current copy rather than duplicates.

Replace changed Nodes and delete Nodes belonging to removed documents.

Retrieval is slow and context keeps growing

Increasing similarity_top_k can improve recall but also increases prompt size, latency, and distraction.

Measure retrieval recall separately from answer quality, rerank when useful, and pass only the context needed. Persist the index instead of rebuilding embeddings at application startup.

The model follows instructions inside a document

Retrieved text is data, not a trusted system message. Separate instructions from evidence.

Prevent retrieved content from granting permissions or authorizing side effects, and limit tools and credentials independently. This belongs to the larger agent security problem.

The honest limitation

LlamaIndex makes the pipeline visible and swappable. It does not make RAG automatically correct.

A vector index can miss a rare term. A chunk can omit an exception. Documents can disagree, parsers can corrupt tables, and models can misread retrieved evidence.

For a small, stable collection, full-text search plus a careful prompt may be cheaper and easier to debug. For exact database questions, query the database rather than embedding every row.

For approvals, payments, or mutations, retrieval should inform a controlled application or agent, not decide permissions.

Use LlamaIndex when you need a composable data layer around unstructured or semi-structured information. Keep the first version small enough to inspect at every stage.

In one breath

• Documents are loaded source material; Nodes are retrievable chunks with text and metadata.
• An Index organises Nodes for lookup. VectorStoreIndex commonly uses embeddings.
• A Retriever selects evidence; a Response Synthesizer writes from it; a Query Engine wraps both.
• Indexing prepares expensive reusable data, while querying retrieves a small context for each question.
• Retrieval is not verification: inspect source_nodes, enforce permissions, and evaluate the whole path.

Quick check

Next

For event-driven orchestration around LlamaIndex agents, see event-driven LlamaIndex Workflows.

For hard-to-parse PDFs, see LlamaParse.