RAG basics — grounding LLMs in your data

A support engineer asks your internal chatbot, “What’s our refund window for enterprise plans?” GPT-5 has never seen your contract templates. It will confidently invent a number. To get a real answer, you need to put the relevant policy paragraph into the prompt at the moment the question is asked. Retrieval-augmented generation (RAG) is the pattern for doing this — finding the right paragraph out of 200,000 documents, in under 100ms, and stuffing it into the prompt before the model sees the question. Retrieval grounds the LLM because the model is told “use only this context” — it reasons over text you control, not over whatever it memorized during training.

The pipeline

Every RAG system has the same shape, split into a slow offline pass and a fast online pass:

That’s it. Every RAG system, no matter how fancy, is doing this. The advanced-RAG lesson covers the upgrades — hybrid retrieval, query rewriting, re-ranking, and Anthropic’s contextual retrieval.

Follow the online path in the figure above one box at a time. The thing to notice: the LLM never sees your documents — only the handful of chunks that retrieval hands it. Almost everything good or bad about a RAG answer is decided before the model is called, in which chunks land in that prompt.

Embeddings: turning text into vectors

An embedding model maps text to a fixed-length list of numbers (a vector) such that semantically similar texts have similar vectors — “refund policy” and “money-back guarantee” land near each other even with no shared words. Common models: OpenAI’s text-embedding-3-large (3072 dims), Cohere’s embed-v4, voyage-3.

Similarity is measured with cosine similarity — the cosine of the angle between two vectors. 1.0 = same direction (similar), 0 = orthogonal (unrelated), -1 = opposite. Cosine is used because most embedding APIs return unit-length vectors, making it as fast as a dot product while ignoring magnitude.

import math

# A mock "embedding" function that produces simple bag-of-words vectors.
# Real embeddings come from a neural model; the SHAPE is the same.
def mock_embed(text):
    text = text.lower()
    vocab = ["python", "sql", "java", "data", "ml", "model", "api",
             "database", "query", "vector", "learn", "train"]
    return [1.0 if w in text else 0.0 for w in vocab]

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb) if na and nb else 0.0

# Index three "documents"
docs = [
    "Python is great for data analysis and ML model training.",
    "SQL queries run against a relational database.",
    "Java is commonly used for enterprise APIs.",
]
index = [(d, mock_embed(d)) for d in docs]

# A user query
query = "How do I train a Python ML model?"
qvec = mock_embed(query)

# Retrieve: score every doc, return top-k
scored = [(d, cosine_similarity(qvec, v)) for d, v in index]
scored.sort(key=lambda x: -x[1])

print("Query:", query)
print("Ranked results:")
for doc, score in scored:
    print(f"  {score:.3f}  {doc}")

# In production, you'd stuff the top-1 or top-3 into the LLM prompt
top_context = scored[0][0]
print(f"\ncontext passed to LLM: {top_context!r}")

Query: How do I train a Python ML model?
Ranked results:
  0.894  Python is great for data analysis and ML model training.
  0.000  SQL queries run against a relational database.
  0.000  Java is commonly used for enterprise APIs.

context passed to LLM: 'Python is great for data analysis and ML model training.'

The query shares python, ml, model, and train with the first doc, so it scores 0.894 while the unrelated docs score 0.000 — and only that top chunk is handed to the LLM. This toy uses literal word overlap; real embedding models capture much more than that, encoding meaning. “How do I train a model?” and “What’s the process for fitting an ML algorithm?” embed near each other even with no shared words — which is the whole reason retrieval finds the right paragraph when the user’s phrasing doesn’t match the document’s.

Chunking: where most RAG dies

Chunking is splitting your documents into the pieces you’ll embed and retrieve — each piece is called a chunk, typically 200-1000 tokens. This is the single biggest source of bad RAG. A few patterns:

• Fixed-size chunks (e.g. 500 tokens with 50-token overlap) — simplest, works surprisingly well, breaks across semantic boundaries.
• Sentence-aware chunks — split on sentences, pack until you hit a token budget. Better preserves meaning.
• Structural chunks — for markdown/code, chunk by headings or functions. Hugely better when documents have structure.
• Late chunking — embed the whole document, then take sliding windows of the resulting representations. Newer technique, often beats other methods.

Common failure modes:

• Chunks too small → context is fragmented, retrieved chunks don’t contain the full answer.
• Chunks too big → embedding becomes a “soup” that doesn’t discriminate; one chunk matches everything.
• Chunks split mid-sentence → retrieved fragments are useless.

Hybrid retrieval: don’t just use embeddings

Pure semantic search misses things. Embeddings are great at meaning but bad at exact matches — they’ll happily retrieve “Python programming” when the user asked about “Python (the snake)”.

Hybrid retrieval combines:

• Dense (semantic) retrieval — embedding cosine similarity.
• Sparse (lexical) retrieval — BM25 or TF-IDF, good at exact keyword matches (acronyms, product names, error codes).

You run both, then merge with Reciprocal Rank Fusion (RRF) or a re-ranker model. Hybrid almost always beats either alone — typically 10-20% recall improvement on real corpora.

Re-ranking

Retrieval gives you top-20-ish candidates. A re-ranker (typically a cross-encoder model like Cohere Rerank or BGE-reranker) scores each candidate against the query and reorders. Cross-encoders are slow (can’t precompute) but accurate. Putting one after retrieval is one of the highest-ROI improvements you can make.

The generation step

Once you have top-k chunks, the prompt to the LLM looks like:

Use only the context below to answer the question. If the answer
isn't in the context, say "I don't know."

Context:
"""
<chunk 1>
---
<chunk 2>
---
<chunk 3>
"""

Question: <user question>
Answer:

Key details:

• Cite sources. Ask for citations to the chunks — easy QA when the model hallucinates.
• Bound the model. Explicit “only use the context” reduces hallucination but doesn’t eliminate it — a model can still misread or ignore context, so RAG is grounding, not a guarantee.
• Watch context length. Stuffing 50 chunks doesn’t help — recent research shows accuracy drops off long before you hit the context window limit. 5-10 chunks is usually the sweet spot.

When RAG isn’t the answer

• The information changes minute-to-minute (stock prices, inventory). Use tool calling to query the live system instead.
• The corpus is small enough to stuff entirely into the context. 100 pages? Just include them. RAG adds complexity you don’t need.
• The task is reasoning, not retrieval. “Summarize the trend in these reports” needs the LLM to read everything, not just chunks.

In one breath

• RAG puts the right paragraph into the prompt at question time so the model reasons over text you control, not its memorized training.
• The pipeline is chunk → embed → store (offline, once) then embed query → retrieve top-k → prompt + generate (online, per question).
• Chunking is where most RAG dies — too small fragments the answer, too big makes one vector that matches everything; sweep the size and measure recall first.
• Hybrid retrieval (dense embeddings + sparse BM25), then a re-ranker, beats pure semantic search by catching exact matches embeddings smudge.
• Bound the model to “use only this context,” cite sources, keep to 5–10 chunks — and remember RAG grounds, it does not guarantee: garbage in, garbage cited.

Quick check

Next

You’ve now seen the foundations: how LLMs generate, how to control output, how to extend them with tools, and how to ground them in your data. From here, the agentic AI track shows how to compose these primitives into systems that act on the world.