RAG Explained: Build a Retrieval-Augmented Generation App with Python

Large Language Models (LLMs) like GPT-4 are incredibly intelligent, but they suffer from a fundamental flaw: they are frozen in time. If you ask an LLM about your company’s internal documentation, a recent news event, or specific proprietary data, it will either hallucinate an incorrect answer or admit it doesn’t know.

You could try fine-tuning the model, but that is expensive, time-consuming, and difficult to update. The modern, efficient solution to this problem is Retrieval-Augmented Generation (RAG).

In this article, you will learn exactly what RAG is, how its architecture works, and how to build a fully functional RAG application from scratch using Python, OpenAI, and ChromaDB.

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation is a technique that gives an AI model the ability to dynamically search an external database for factual information before answering a question.

Think of a standard LLM taking a closed-book exam. It has to rely solely on the facts it memorized during its initial training. RAG, on the other hand, turns it into an open-book exam. When you ask a question, the system first retrieves the relevant textbook pages, reads them, and then synthesizes an accurate answer based on those specific pages.

By grounding the AI’s responses in retrieved facts, RAG drastically reduces hallucinations, allows you to use up-to-date information without retraining the model, and enables you to cite your sources.

RAG vs. Fine-Tuning

A common question among developers is whether to use RAG or fine-tune an existing model. While both customize AI behavior, they serve different purposes:

Feature	Retrieval-Augmented Generation (RAG)	Fine-Tuning
Primary Use Case	Adding new knowledge, facts, or context.	Changing the model’s style, tone, or format.
Data Updates	Instant. Just add or delete a document in the database.	Slow. Requires retraining the model.
Cost	Low (only pay for API calls and database hosting).	High (requires high-end GPUs and compute time).
Hallucination Risk	Low (grounded in specific retrieved context).	Moderate (models can still invent facts).
Source Tracing	Yes (can link exactly to the retrieved document).	No (knowledge is baked into the weights).

For 95% of business applications—such as customer support bots, internal knowledge bases, and document analyzers—RAG is the superior choice.

The RAG Architecture Explained

A standard RAG pipeline operates in two distinct phases: Data Ingestion and Retrieval/Generation.

Phase 1: Data Ingestion (Preparing the Knowledge)

Before you can search your data, the AI needs to understand it.

1. Document Loading: You load raw text (PDFs, websites, markdown files).
2. Chunking: The text is split into smaller, manageable pieces (e.g., 500 words each). LLMs have context limits, so we only want to retrieve the most relevant paragraphs, not a whole book.
3. Embedding: A specialized AI model converts these text chunks into vector embeddings—long lists of numbers that capture the semantic meaning of the text.
4. Vector Database: These numerical vectors are stored in a specialized database (like ChromaDB or FAISS) built to search through numbers extremely fast.

Phase 2: Retrieval and Generation (Answering the User)

1. Query Embedding: When a user asks a question, the question is converted into a vector using the same embedding model.
2. Vector Search (Retrieval): The database calculates the mathematical distance between the question’s vector and the document vectors. It returns the top chunks that are most similar to the question.
3. Prompt Augmentation: The retrieved chunks are injected into the prompt alongside the user’s original question.
4. Generation: The LLM reads the augmented prompt and generates a final, accurate answer.

Build Your First RAG App in Python

Let’s build a minimal, working RAG application. We will use OpenAI for both the embeddings and the text generation, and ChromaDB as our local vector database.

Prerequisites

You need Python 3.8+ installed and an active OpenAI API key.

Create a new directory and install the required packages:

pip install chromadb openai python-dotenv

Step 1: Set Up the Environment

Create a .env file in your project directory to store your API key safely:

OPENAI_API_KEY=your_actual_api_key_here

Step 2: The Python Code

Create a file named rag_app.py and add the following code. We will build this without a massive framework like LangChain so you can see exactly how the mechanics work under the hood.

import os
import chromadb
import chromadb.utils.embedding_functions as embedding_functions
from openai import OpenAI
from dotenv import load_dotenv

# 1. Load environment variables and initialize clients
load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")

openai_client = OpenAI(api_key=api_key)
chroma_client = chromadb.Client()

# 2. Configure the Embedding Function
# This converts our text into numerical vectors using OpenAI's embedding model
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=api_key,
    model_name="text-embedding-3-small"
)

# 3. Create a Vector Database Collection
collection = chroma_client.create_collection(
    name="company_knowledge_base", 
    embedding_function=openai_ef
)

# 4. Ingest Data (Phase 1)
# In a real app, you would load these from PDFs or a database.
documents = [
    "Xray AI Lab was founded in 2024 to provide high-quality AI education.",
    "Our flagship product is the 'Python for AI Bootcamp', which costs $199.",
    "The company headquarters is located in Austin, Texas.",
    "Refunds are only available within 14 days of purchase."
]

# We need unique IDs for each document chunk
ids = [f"doc_{i}" for i in range(len(documents))]

print("Embedding and storing documents...")
collection.add(
    documents=documents,
    ids=ids
)
print("Documents stored successfully!\n")

# 5. Retrieval & Generation (Phase 2)
def ask_rag_bot(question):
    # Retrieve the most relevant documents based on the question
    results = collection.query(
        query_texts=[question],
        n_results=2 # Fetch the top 2 most relevant chunks
    )
    
    retrieved_context = results['documents'][0]
    context_string = "\n".join(retrieved_context)
    
    # Construct the augmented prompt
    prompt = f"""
    You are a helpful customer support assistant for Xray AI Lab.
    Answer the user's question using ONLY the context provided below.
    If you do not know the answer based on the context, say "I don't have that information."
    
    Context:
    {context_string}
    
    Question: {question}
    """
    
    # Generate the answer using an LLM
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    
    return response.choices[0].message.content

# 6. Test the RAG Application
if __name__ == "__main__":
    user_question = "How much does the Python bootcamp cost, and where is the company based?"
    
    print(f"User: {user_question}")
    print("Agent is thinking (and retrieving)...")
    
    answer = ask_rag_bot(user_question)
    
    print(f"\nAI Answer: {answer}")

Understanding the Code

When you run python rag_app.py, the following happens:

1. ChromaDB creates an in-memory vector database.
2. The collection.add() function sends our four sentences to OpenAI, converts them to vectors (embeddings), and saves them.
3. When we call ask_rag_bot(), ChromaDB compares the question against the stored vectors and retrieves the sentences about the bootcamp price and headquarters.
4. We inject those specific sentences into the GPT-4 prompt.
5. The model outputs a perfect, factually correct answer based on our proprietary data.

Next Steps for Improving Your RAG Pipeline

The script above is a perfectly functional prototype, but production RAG systems require more advanced engineering:

• Advanced Document Parsers: Use libraries like PyMuPDF or Unstructured to extract text cleanly from complex PDFs and Word documents.
• Intelligent Chunking: Instead of splitting text arbitrarily by character count, use recursive chunking or semantic chunking to ensure paragraphs and thoughts remain whole.
• Hybrid Search: Combine vector search (for semantic meaning) with traditional keyword search (BM25) to retrieve exact names, acronyms, or ID numbers more reliably.

Building a RAG application demystifies how modern AI assistants actually work in the enterprise. By separating knowledge storage from reasoning, you gain total control over what your AI knows and how it answers.

---

References and Further Reading

• Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. arXiv:2005.11401
• OpenAI Documentation: Embeddings
• ChromaDB Official Documentation: Getting Started