AI Tools 7 min read ·

How to Build a RAG Application Step by Step

We earn commissions when you shop through the links below.

If you’ve been trying to figure out how to build a RAG application step by step, you’re not alone. Retrieval-Augmented Generation has gone from research paper to production staple fast, and for good reason — it lets you give any LLM access to your own data without fine-tuning. In this guide I’ll walk you through the entire pipeline: loading documents, chunking, embedding, storing vectors, retrieving context, and finally generating answers. We’ll write real Python code and keep things practical.

What Is RAG and Why Does It Matter?

RAG stands for Retrieval-Augmented Generation. Instead of relying solely on what an LLM memorized during training, you retrieve relevant chunks from your own knowledge base at query time and inject them into the prompt. The model then generates an answer grounded in that context.

This matters because:

  • LLMs hallucinate less when given factual context
  • Your data stays private — you don’t need to retrain or fine-tune
  • You can update the knowledge base without touching the model
  • It works with any LLM that accepts a system prompt

The RAG Pipeline at a Glance

Before diving into code, here’s the high-level flow:

  1. Load — ingest your documents (PDFs, Markdown, web pages, databases)
  2. Chunk — split documents into smaller, semantically meaningful pieces
  3. Embed — convert each chunk into a vector using an embedding model
  4. Store — persist vectors in a vector database
  5. Retrieve — at query time, embed the question and find the top-k similar chunks
  6. Generate — pass retrieved chunks + question to an LLM and return the answer

That’s the whole thing. Let’s build it.

Prerequisites

You’ll need Python 3.11+, an OpenAI API key, and the following packages:

pip install openai langchain langchain-community langchain-openai \
    chromadb tiktoken pypdf

I’m using Cursor as my editor throughout this project — its inline AI assistance speeds up writing boilerplate for RAG pipelines dramatically.

Step 1 — Load Your Documents

LangChain has document loaders for practically every format. Here we’ll load a directory of PDFs:

from langchain_community.document_loaders import PyPDFDirectoryLoader

loader = PyPDFDirectoryLoader("./docs")
raw_documents = loader.load()

print(f"Loaded {len(raw_documents)} pages")

Each document object contains page_content (text) and metadata (source file, page number, etc.). Keep the metadata — it’s useful for citations later.

Step 2 — Chunk the Documents

Chunking is where most people make mistakes. Too large and you waste context window space with irrelevant text. Too small and you lose semantic coherence. A good starting point is 512 tokens with a 50-token overlap:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    length_function=len,
    separators=["\n\n", "\n", " ", ""]
)

chunks = splitter.split_documents(raw_documents)
print(f"Created {len(chunks)} chunks")

RecursiveCharacterTextSplitter tries to split on paragraph breaks first, then newlines, then spaces — preserving sentence structure as much as possible.

Step 3 — Embed the Chunks

We convert each chunk into a dense vector. OpenAI’s text-embedding-3-small is cheap, fast, and performs well for most use cases:

from langchain_openai import OpenAIEmbeddings
import os

os.environ["OPENAI_API_KEY"] = "your-api-key-here"

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

The embedding model is called automatically when you add documents to the vector store in the next step — you don’t need to call it manually.

Step 4 — Store in a Vector Database

For local development, ChromaDB is excellent — it’s persistent, fast, and needs no external service. For production, you’ll want something like Pinecone, Qdrant, or pgvector.

from langchain_community.vectorstores import Chroma

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

print("Vector store created and persisted.")

Run this once to build the index. After that you can load it without re-embedding:

vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings
)

Step 5 — Build the Retriever

The retriever takes a query, embeds it, and returns the top-k most similar chunks:

retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 5}
)

You can also use search_type="mmr" (Maximum Marginal Relevance) to get diverse results instead of the five most similar — useful when your chunks are repetitive.

Step 6 — Build the Generation Chain

Now we connect the retriever to an LLM. Here’s a complete RAG chain using LangChain’s expression language (LCEL):

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

llm = ChatOpenAI(model="gpt-4o", temperature=0)

prompt = ChatPromptTemplate.from_template("""
You are a helpful assistant. Answer the question based only on the context below.
If the answer is not in the context, say "I don't have enough information to answer that."

Context:
{context}

Question: {question}

Answer:
""")

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Run a query
answer = rag_chain.invoke("What are the main product features?")
print(answer)

That’s a fully functional RAG pipeline. The chain automatically retrieves context, formats it, builds the prompt, calls the LLM, and returns a clean string.

Adding Source Citations

One of the most requested features — showing users which documents the answer came from. Here’s how to return both the answer and sources:

from langchain_core.runnables import RunnableParallel

rag_chain_with_sources = RunnableParallel(
    {"context": retriever, "question": RunnablePassthrough()}
).assign(
    answer=(
        lambda x: {"context": format_docs(x["context"]), "question": x["question"]}
        | prompt
        | llm
        | StrOutputParser()
    )
)

result = rag_chain_with_sources.invoke("What are the main product features?")
print(result["answer"])
for doc in result["context"]:
    print(f"Source: {doc.metadata.get('source', 'unknown')}, Page: {doc.metadata.get('page', 'N/A')}")

Deploying Your RAG Application

Once you’re happy with the pipeline locally, you need somewhere to run it. I typically wrap the chain in a FastAPI app and deploy it on DigitalOcean — their App Platform handles container deployments cleanly with managed databases if you want to switch to pgvector later.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class QueryRequest(BaseModel):
    question: str

@app.post("/ask")
def ask(request: QueryRequest):
    answer = rag_chain.invoke(request.question)
    return {"answer": answer}

Run with uvicorn main:app --reload locally, then containerize with Docker for deployment.

Common Pitfalls and How to Avoid Them

After building several RAG applications, here are the issues I hit most often:

  • Bad chunking strategy — don’t just use character splits blindly. For structured docs like code or tables, use format-aware splitters.
  • Not enough retrieved chunks — start with k=5 or k=10. Retrieval is cheap; being too conservative hurts answer quality.
  • Ignoring metadata filtering — if your docs span multiple topics or time periods, add metadata filters to your retriever so irrelevant chunks don’t pollute context.
  • No evaluation — use RAGAS or a simple LLM-as-judge setup to measure faithfulness and answer relevancy. Without metrics you’re flying blind.
  • Embedding model mismatch — always use the same embedding model for indexing and querying. Mixing models produces garbage retrieval.

Next Steps

Now that you know how to build a RAG application step by step, there are several directions to take it further:

  • Hybrid search — combine vector similarity with BM25 keyword search for better recall on exact terms
  • Re-ranking — use a cross-encoder model to rerank retrieved chunks before passing to the LLM
  • Query rewriting — have the LLM rephrase the user’s question before retrieval to improve semantic matching
  • Agentic RAG — let the model decide when to retrieve and what to search for, rather than always retrieving on every query

If you want to go deeper on the theory and advanced patterns, there are solid courses on Udemy covering LangChain and RAG architecture in detail.

Wrapping Up

Knowing how to build a RAG application step by step is now a core skill for anyone shipping AI features. The pipeline is straightforward once you’ve done it once: load, chunk, embed, store, retrieve, generate. The complexity lives in tuning each stage for your specific data and use case — but the foundation you built here scales to production.

Start with the code above, measure your retrieval quality early, and iterate from there. The best RAG systems aren’t the ones with the fanciest architecture — they’re the ones with clean data and well-tuned chunking.