Systems & Architecture Aliases: RAG, Retrieval-Augmented Generation

Retrieval-Augmented Generation

An architectural pattern that combines parametric memory (pre-trained neural networks) with non-parametric memory (external document indexes) to generate factual, verifiable, and dynamically updatable responses.

In One Sentence

Instead of relying solely on static weights memorized during training, a RAG system dynamically queries an external database at inference time to inject relevant context directly into the model's prompt window.

Intuition

Imagine taking a highly technical exam. You have two options:

Closed-Book (Parametric Only): You must answer entirely from memory. If facts change after you study, or if you need to cite exact sources, you are highly prone to guessing or "hallucinating" plausible-sounding details.
Open-Book (Retrieval-Augmented): You are given a search engine or a library of textbooks. For every question, you search for the exact page, read the relevant paragraphs, and synthesize a precise, sourced answer.

RAG brings this "open-book" paradigm to language models. It decouples knowledge storage (which is kept in an easily updatable external index) from reasoning and synthesis (which is handled by the model's pre-trained parameters).

Key Advantages

Zero-Shot Knowledge Updates

Update the database, and the model instantly knows new facts without expensive retraining.

Provenance & Citations

Every generated claim can be mapped directly back to the source document retrieved.

Reduced Hallucinations

Grounding the generation process in retrieved documents dramatically lowers factual errors.

Formal Definition

In their foundational paper, Lewis et al. (2020) formalized RAG as a sequence-to-sequence model that conditions generation on a latent variable representing retrieved documents.

Let $x$ be the input query, $z$ be a retrieved document from a corpus $\mathcal{D}$, and $y = (y_1, \dots, y_N)$ be the generated sequence of tokens. The system consists of two main components:

Retriever $p_\eta(z|x)$: A dense vector search model (e.g., Dense Passage Retrieval) that returns a probability distribution over the top-$k$ documents most relevant to $x$.
Generator $p_\theta(y_i | x, z, y_{: A parametric sequence-to-sequence model (e.g., BART, GPT) that generates tokens conditioned on both the query \(x$ and the retrieved document $z$.

Two Formulations of RAG

1. RAG-Sequence Model

The model retrieves a set of documents and uses the same document to generate the entire output sequence. The probabilities are marginalized across the top-$k$ documents:

$$P_{\text{RAG-Seq}}(y|x) \approx \sum_{z \in \text{top-}k} p_\eta(z|x) \prod_{i=1}^N p_\theta(y_i | x, z, y_{

2. RAG-Token Model

The model can retrieve and condition on different documents for each token. This allows the generator to synthesize information from multiple sources step-by-step:

$$P_{\text{RAG-Tok}}(y|x) \approx \prod_{i=1}^N \sum_{z \in \text{top-}k} p_\eta(z|x) p_\theta(y_i | x, z, y_{

When to Use vs. When Not to

Use When...

• Dynamic Knowledge: Your domain data changes frequently (e.g., news, stock prices, code repositories).

• Factual Precision: The cost of hallucination is high (e.g., medical advice, legal search, customer support).

• Massive Corpora: You have millions of documents that cannot fit into even large context windows.

Do Not Use When...

• Ultra-Low Latency: Retrieval steps (vector search, document fetching) add network and compute overhead.

• Purely Creative Tasks: Tasks that require abstract reasoning, style mimicry, or brainstorming without factual constraints.

• Static / Small Contexts: If the entire corpus is small enough to fit inside a modern long-context window (e.g., < 100k tokens), direct prompting may perform better.

History & Evolution

May 2020
Foundational RAG Paper

Lewis et al. publish "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". They introduce the end-to-end differentiable RAG framework, combining a dense retriever (DPR) with a seq2seq generator (BART) to set new state-of-the-art results on open-domain QA.

October 2022
Interleaved Reasoning & Acting (ReAct)

Yao et al. introduce "ReAct: Synergizing Reasoning and Acting in Language Models". They demonstrate how LLMs can dynamically decide when to interact with external environments (such as a Wikipedia API) to retrieve information on-the-fly, expanding the static retrieve-then-read paradigm into an agentic loop.

Interactive Walkthrough

Explore how a RAG pipeline processes a user query step-by-step. Select a sample query below to trace its path through retrieval to final generation.

Select Query:

1. ENCODE QUERY

2. RETRIEVE

3. AUGMENT PROMPT

4. GENERATE

Step 1 of 4

Pseudocode

rag_pipeline.py Python-flavored

import numpy as np class RAGPipeline: def __init__(self, retriever_model, generator_model, document_index): self.retriever = retriever_model self.generator = generator_model self.index = document_index # Vector database containing document embeddings def retrieve(self, query: str, k: int = 3): # 1. Encode query into dense vector space query_vector = self.retriever.encode(query) # 2. Perform vector search over external index top_k_docs = self.index.search(query_vector, limit=k) return top_k_docs def generate(self, query: str, retrieved_docs: list) -> str: # 3. Format context from retrieved documents context = "\n\n".join([doc["text"] for doc in retrieved_docs]) # 4. Construct augmented prompt prompt = f"""Answer the question based only on the context provided. Context: {context} Question: {query} Answer:""" # 5. Generate response via autoregressive LLM response = self.generator.generate(prompt) return response def run(self, query: str) -> str: docs = self.retrieve(query, k=2) answer = self.generate(query, docs) return answer

Common Variants

Naive RAG

The standard Retrieve-then-Read pipeline. It retrieves documents based on the raw query, appends them to the context window, and generates the output in a single forward pass.

Advanced RAG

Introduces pre-retrieval strategies (like query rewriting or expansion) and post-retrieval strategies (like reranking via cross-encoders) to optimize context quality before generation.

Agentic / Iterative RAG

The model dynamically decides when to retrieve information during generation, using multi-step reasoning traces (e.g., ReAct) to resolve complex, multi-hop queries.

Representative Flash Papers

Explore foundational models and concepts in our corpus that relate to the development of parametric and non-parametric architectures.

BERT Explained

BERT Explained: Deep Bidirectional Transformers

Bidirectional representations from Transformers. BERT models often serve as the dense encoder backbone for neural retrievers (such as DPR) used in modern RAG systems.

Few-Shot Learners

Language Models are Few-Shot Learners

Demonstrates the immense parametric memory and reasoning capabilities of massive language models, highlighting the boundary where non-parametric retrieval becomes necessary for dynamic knowledge.

References & Grounding

[1]
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al., 2020. arXiv:2005.11401.
Introduced the RAG formulation, combining parametric seq2seq models with dense vector indexes of Wikipedia.

[2]
ReAct: Synergizing Reasoning and Acting in Language Models
Yao et al., 2022. arXiv:2210.03629.
Explored interleaving reasoning steps with actions (retrieval API calls) to solve complex open-domain QA tasks.