In LlamaIndex, what are nodes and query engines, and how is RAG exposed as a tool to an agent?

How to think about it

A node is a retrievable piece of source content, usually a chunk of a document with metadata and relationships. A query engine is the component that retrieves relevant nodes and uses them to produce an answer. To expose retrieval-augmented generation, or RAG, to an agent, I wrap the query engine in a QueryEngineTool; the agent can then decide whether to call that knowledge source, just as it might call a calculator or database tool.

Why these objects exist

Suppose a company has a 120-page employee handbook. Asking an LLM about parental leave directly is unreliable because the model may not have the handbook, may have seen an older policy, or may simply produce a plausible answer. RAG solves that by looking up relevant source text at question time and placing it in the generation prompt.

LlamaIndex breaks that job into useful layers.

A document is the source object loaded from a file, web page, database row, or another connector. A node is the smaller unit created from that document for indexing and retrieval. In a simple text pipeline, one document becomes many nodes. A node is often a TextNode, although nodes can represent more structured content too.

A node contains more than text:

• Its text might say, “Employees may carry over up to five unused parental-leave days.”
• Its metadata might contain the page number, department, document version, effective date, and access-control label.
• Its relationships might point to the source document, the previous and next node, or a parent section.

That extra information matters. If an answer needs a citation, page metadata makes one possible. If a policy is split across two adjacent chunks, relationships can help preserve context. If the company has separate policies for contractors and employees, metadata can support filtering.

An index is the data structure that makes nodes searchable. With a vector index, LlamaIndex converts each node into an embedding, which is a numerical representation intended to place semantically similar text near one another. A question about “unused leave carried into next year” can therefore find a node containing “carry-over of parental days,” even though the wording differs.

The index is not the answer generator. It helps find candidate nodes.

A query engine is the answering interface around retrieval and synthesis. It normally contains a retriever, which selects relevant nodes, and a response synthesizer, which uses those nodes to formulate a response. Depending on the underlying index and configuration, retrieval may use vectors, keywords, metadata filters, SQL, or a hybrid of several methods.

That distinction is worth stating clearly:

Common misconception: a query engine is not just a vector database, and it is not an agent. It is a query-time pipeline that receives a question, retrieves context, and returns a response.

The agent appears one layer higher.

An agent is an LLM-driven control loop that can choose actions, call tools, inspect results, and continue or stop. QueryEngineTool adapts a LlamaIndex query engine to the tool interface expected by an agent. The tool exposes a name and description, and the agent uses those to decide when the tool is appropriate.

A concrete handbook example

Imagine the handbook has been split into 800 nodes. Each node is roughly 300 tokens and carries metadata such as:

document: employee_handbook
page: 47
section: parental_leave
effective_from: 2026-01-01
audience: employee

A basic LlamaIndex setup could look like this:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.tools import QueryEngineTool

documents = SimpleDirectoryReader("handbook").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine(similarity_top_k=4)

handbook_tool = QueryEngineTool.from_defaults(
    query_engine=query_engine,
    name="company_handbook",
    description=(
        "Search the current employee handbook for policies about leave, "
        "benefits, working hours, and workplace rules. Do not use this for "
        "live payroll balances or personal employee records."
    ),
)

tools_for_agent = [handbook_tool]

The important line is not the index construction. It is the wrapping step:

handbook_tool = QueryEngineTool.from_defaults(...)

The agent receives tools_for_agent along with any other tools. The exact agent class can vary across LlamaIndex versions and agent designs, but the conceptual contract stays the same.

A user asks:

How many unused parental-leave days can I carry into 2027?

The sequence is:

1. The agent reads the tool name and description.
2. It decides that the handbook is relevant.
3. It calls the tool with the user’s question.
4. The query engine retrieves, for example, four relevant nodes.
5. The response synthesizer combines those nodes into a grounded answer.
6. The tool result is returned to the agent.
7. The agent gives the final response, possibly with a page citation.

The query engine might retrieve a node from page 47 saying that five days can be carried over. It might also retrieve a nearby node saying that the rule does not apply after termination. The synthesizer needs both pieces if the user’s situation includes a leaving date.

The agent does not need to know how embeddings, chunking, or vector search work. It only needs a clear tool contract. That separation is one of the practical advantages of this pattern.

What makes the tool description important

The description is a routing instruction, not decorative documentation. An agent with five tools needs to distinguish them. A useful description says what the tool knows and, just as importantly, what it does not know.

For example, “Search company information” is vague. “Search the current employee handbook for leave and benefits policies; do not use it for live payroll balances” gives the agent a usable boundary.

That boundary is not a security control. If the handbook contains confidential information, the application must enforce authorization before retrieval and apply user-specific metadata filters. An LLM promising not to access payroll data is not an access-control system. It is a sentence with excellent intentions.

The senior-level nuance

RAG as a tool is useful when the agent has a genuine choice among knowledge sources. For example, an HR agent might have:

Tool	Best for
company_handbook	Stable policy text
payroll_system	A specific employee’s current balance
calendar_tool	Dates, holidays, and scheduling
ticket_search	Prior support cases

A question about carry-over policy should use the handbook. A question about “my remaining balance” should use the payroll system. A query engine alone cannot make that routing decision reliably; the agent can.

But an agent is not automatically better. If every request goes to the same handbook, calling an agent first adds an LLM decision step, cost, latency, and another source of nondeterminism. A direct query_engine.query(...) call is often simpler and easier to test. I would use RAG as a tool when tool selection or multi-step work creates real value, not merely because the word “agent” is fashionable.

The retrieval pipeline also has its own trade-offs. If chunks are too large, a top result may contain several unrelated policies, diluting the relevant passage. If chunks are too small, an exception may be separated from the rule it qualifies. Four retrieved nodes are not inherently better than two or eight; the right value depends on chunk size, document structure, retrieval quality, and the context budget of the model.

Versioning is another production concern. If the handbook is replaced but the old nodes remain in the index, the agent may retrieve conflicting policies. The first symptom may be an answer that cites page 47 correctly but gives the retired five-day rule instead of the current rule. I would store document version and effective date as metadata, filter retrieval to the current policy, and test questions that deliberately mention dates and exceptions.

A different failure appears when the agent never calls the tool. At 3 a.m., the logs show a confident answer with zero retrieval requests. That usually points to a poor tool description, an agent-routing problem, or an overly broad system prompt. I would log tool-selection decisions, tool inputs, retrieved node identifiers, scores where available, and the final answer source. Without those traces, “the RAG system is wrong” is not a diagnosis.

What they’ll ask next

Does a query engine always use vector search?

No. A query engine is an answering interface, not a synonym for vector search. It can sit on top of a vector index, a keyword retriever, a SQL query engine, a property graph, or a composed retrieval pipeline. The underlying retriever determines how relevant context is found.

Why not give the nodes directly to the agent instead of using a query-engine tool?

Because the query engine packages retrieval and synthesis behind a stable interface. The agent supplies a natural-language question rather than managing embeddings, top-result selection, context formatting, and response generation itself. Direct node access can be appropriate for custom workflows, but it pushes those responsibilities into the agent or application code.

How would you evaluate this system?

I would separate routing, retrieval, and answer quality. First, measure whether the agent selects the handbook for handbook questions. Next, check whether the correct nodes appear in the retrieved set. Finally, evaluate whether the response faithfully answers from those nodes, handles conflicting versions, and declines when the evidence is missing. A fluent answer is not proof that retrieval worked.

Say this in the interview

“LlamaIndex nodes are metadata-rich, retrievable units of source content; a query engine retrieves those nodes and synthesizes an answer, and QueryEngineTool exposes that pipeline to an agent so the agent can choose when to use the knowledge source as part of its tool-calling loop.”