Hands-On Lab · Applied GenAI

Scaffolding a RAG Powered AI Assistant

A guided walkthrough of the whole build, from the modular scaffold and tech stack, through every route and service (PDF extraction, embeddings, re-ranking, the full RAG pipeline, RAGAS evaluation), to Docker packaging, so you can follow along and build your own.

Part 1: scaffold → stack → architecture → upload.py Part 2: topics · suggestions · query · evaluate · TTS routes → all services → Docker Reference repo: Anjali-k27/Single-Source-Retrieval

⭐ Trick the insight that makes it easy ⚠ Pitfall a common mistake 💡 Tip exam / interview aside [+] added, not in sources [!] source conflict flagged

The one-page revision sheet

Telegraphic and hand-copyable. Everything here is expanded with full explanation in Section 2.

---

The full lesson, taught from the ground up

The problem we're solving

You have a 200-page technical PDF and a question buried somewhere inside it. You paste the whole thing into a chatbot, but it's too long, so the model forgets the start by the time it reads the end, and worse, it sometimes invents an answer that sounds right but appears nowhere in your document. That confident-but-wrong behavior is called hallucination, and it's the exact failure this project is built to prevent.

This project builds a "upload your PDF, ask a question, get an answer that actually comes from that PDF" assistant, with a twist: you can ask by typing or speaking, and it can read the answer back to you. Under the hood it's a full-stack application with a proper production-grade structure, not a single throwaway notebook. This is a deliberate choice: modular programming (splitting code into small, single-purpose files) is what separates "I trained a model" from "I shipped a model."

The tech stack, layer by layer

Before touching code, name every tool and why it's there. Think of the app as a restaurant: the frontend is the dining room the customer sees, the backend is the kitchen where the work happens, and the LLM is the master chef doing the actual reasoning.

What RAG actually does

RAG (Retrieval-Augmented Generation) means: before the model answers, first go fetch the relevant passages, then make the model answer using only those passages. The "augmented" part is the retrieved text you hand to the model as context.

Analogy, the open-book exam. A pure LLM is a student answering from memory, fluent, but prone to making things up under pressure. RAG turns it into an open-book exam: the student first flips to the right pages (retrieval), then writes the answer while looking at them (generation). The answer is now tied to the source, not to whatever the model half-remembers from training.

Why the code is split into so many files

Modular programming means each file does exactly one job, so a change in one place can't silently break another. The alternative, one giant script.py, works on your laptop and collapses the moment a teammate touches it.

Analogy, a well-run kitchen. You don't want one person chopping, frying, plating, and taking orders at once. You want stations. In this project the stations are folders: routes/ takes the orders, services/ cooks, config.py is the recipe card of settings, and models.py is the standard order-ticket format everyone agrees on.

The __init__.py trick

Every folder that holds Python code gets an empty file named __init__.py. This one-liner (well, zero-liner) is what lets you write from app.routes import upload from elsewhere. Without it, Python doesn't treat the folder as an importable package, and your imports fail before the app even starts.

Docker and GitHub Actions, the deployment pair

Docker packages your entire app plus its exact dependencies into a sealed box (a container) that runs identically on any machine. The story is the universal one: you build something that works locally, zip it, send it to your manager, and it dies on their machine, they have Python 2.1, you built on 3.11. That version mismatch is the "dependency bottleneck."

Analogy, the shipping container. Before standardized containers, moving cargo between truck, ship, and train was chaos. A container fixed everything: same box, every vehicle, every port. Docker does that for software, the same box runs on your laptop, your teammate's, and the production server.

GitHub Actions is the automation belt. It watches your repository and, on every push, runs a sequence of steps for you, this is CI/CD (Continuous Integration / Continuous Deployment). In this project the workflow lints the code, runs the tests, builds both Docker images, and has a deploy step ready to wire up. You change code, push, and the belt takes over.

The project tree (from the actual repo)

This is the real structure that got pushed to the reference repo. The transcript built the backend piece by piece; here it is assembled.

# root/
├── backend/
│   ├── app/
│   │   ├── main.py                 # FastAPI app + CORS + routers
│   │   ├── config.py               # Pydantic settings (env vars)
│   │   ├── models.py               # request/response schemas
│   │   ├── routes/
│   │   │   ├── upload.py           # POST /api/upload
│   │   │   ├── query.py            # POST /api/query
│   │   │   └── evaluate.py         # POST /api/evaluate
│   │   ├── services/
│   │   │   ├── document_processor.py  # PDF → chunks → FAISS
│   │   │   ├── embeddings.py          # Google embedding-001 + retry
│   │   │   ├── rag_pipeline.py        # full RAG orchestration
│   │   │   ├── reranker.py            # Cross-Encoder re-ranking
│   │   │   └── evaluator.py           # RAGAS metrics + JSONL logging
│   │   └── utils/retry.py          # tenacity decorators
│   └── tests/test_api.py
├── frontend/
│   └── src/                        # app/ components/ hooks/ lib/ types/
├── .github/workflows/main.yml      # CI/CD pipeline
├── docker-compose.yml
└── README.md

Inside the backend app

The instructor covered three foundational files in depth (main.py, config.py, models.py) and one full route (upload.py). Everything below reconstructs the taught logic. Code shown is faithful to what was described; anything not stated verbatim in a source is marked [+].

main.py, the entry point

The first file that runs. It creates the FastAPI application, adds middleware, and wires up every route. When a request hits the server, this is the switchboard that decides where it goes.

Three responsibilities, in order:

1. Create the app. Make a FastAPI() object with a title, version, and docs URL. FastAPI auto-generates interactive API docs at /docs (Swagger).
2. Add CORS middleware. Explained below, this is the security gate between frontend and backend.
3. Include the routers. Each route file exposes a router; main.py attaches them so their endpoints go live. It also defines a /health check that returns {"status":"ok"}.

# app/main.py  [+ illustrative, matches the described flow]
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.config import get_settings
from app.routes import upload, query, evaluate

settings = get_settings()
app = FastAPI(title="My Learning AI Assistant", version="1.0", docs_url="/docs")

app.add_middleware(
    CORSMiddleware,
    allow_origins=[settings.frontend_url],   # e.g. http://localhost:3000
    allow_methods=["*"], allow_headers=["*"],
)

app.include_router(upload.router,   prefix="/api")
app.include_router(query.router,    prefix="/api")
app.include_router(evaluate.router, prefix="/api")

@app.get("/health")
def health():
    return {"status": "ok"}

What is CORS, in plain words? Your browser has a built-in security rule: a page loaded from one origin (say localhost:3000, the frontend) is normally not allowed to call a server at a different origin (localhost:8000, the backend). CORS (Cross-Origin Resource Sharing) is the mechanism where the backend explicitly says "I permit that frontend to talk to me." The middleware stamps that permission on responses.

config.py, the single source of settings

One file that holds every configurable value: API keys, model names, URLs, file paths, and retrieval numbers. Change a setting here, not scattered across the codebase.

It uses Pydantic Settings, a class that declares each setting with its type and pulls values from environment variables. Declaring api_key: str means "this must be a string", if something else shows up, it's rejected early. That's the "define the format the backend expects" idea.

# app/config.py  [+ illustrative]
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    # --- secrets (loaded from .env) ---
    openrouter_api_key: str
    google_api_key: str
    # --- models ---
    chat_model: str   = "google/gemini-2.5-flash"
    openrouter_url: str = "https://openrouter.ai/api/v1"
    embedding_model: str = "models/embedding-001"
    reranker_model: str  = "cross-encoder/ms-marco-MiniLM-L-6-v2"
    # --- two-stage retrieval ---
    top_k: int        = 15   # stage 1: bi-encoder recall
    rerank_top_n: int = 5    # stage 2: cross-encoder keeps top 5
    # --- chunking ---
    chunk_size: int    = 1000
    chunk_overlap: int = 150
    # --- storage + server ---
    faiss_index_path: str = "data/faiss_index"
    eval_log_path: str    = "logs/ragas_eval.jsonl"
    frontend_url: str     = "http://localhost:3000"

    class Config:
        env_file = ".env"

def get_settings() -> Settings:
    return Settings()   # create once, reuse the object everywhere

models.py, the shape of every request and response

Defines the exact structure of data flowing in and out, as Pydantic classes. It's the contract between frontend and backend: both sides agree on field names and types up front.

Analogy, the standardized order ticket. In a busy kitchen, every order is written on the same ticket format. No one guesses what "the usual" means. models.py is that ticket, an UploadResponse always has a document_id, a chunks count, and a message, so nobody downstream has to guess.

# app/models.py  [+ illustrative, fields per the transcript]
from pydantic import BaseModel

class UploadResponse(BaseModel):
    message: str
    document_id: str
    chunks: int
    description: str

class QueryRequest(BaseModel):
    document_id: str
    question: str

class EvaluationRequest(BaseModel):
    question: str
    answer: str
    context: list[str]
    ground_truth: str        # the 4 inputs RAGAS needs

Two-stage retrieval, the heart of "advanced" RAG

Instead of trusting one search to be both fast and accurate, split it: a cheap search casts a wide net, then an expensive scorer carefully picks the best few. This is the single most important design idea in the whole pipeline.

Analogy, hiring. A recruiter first screens 100 résumés fast and shortlists 15 (that's the bi-encoder, quick and approximate). Then a hiring manager interviews those 15 carefully and picks the top 5 (that's the cross-encoder, slow but precise). You'd never interview all 100, and you'd never hire off a 5-second résumé glance alone.

Chunking and overlap

A chunk is a bite-sized slice of the document; overlap means neighboring chunks share a bit of text at the seam. You chunk because you can't embed a whole book as one vector, and you overlap so a sentence split across the boundary isn't lost.

An example: chunk one ends "…the exam was difficult, we tried to attempt as much as we can, but the instructions". If chunk two starts fresh after that, the thought is decapitated. With overlap, the tail of chunk one ("…but the instructions") reappears at the head of chunk two, so whichever chunk gets retrieved still carries the full context.

upload.py, the first working route

The endpoint that receives a PDF, validates it, turns it into searchable vectors, and hands back an ID. This is where the abstract pipeline becomes a real HTTP call, walked through end to end.

The flow inside POST /api/upload:

1. Guard the size. Max is hard-coded at 50 MB. Convert with 50 × 1024² bytes. Over the limit → raise HTTP 413 (Request Entity Too Large).
2. Guard emptiness. Empty file → HTTP 400 with a clear message.
3. Ingest. Read the PDF bytes, hand off to DocumentProcessor.ingest_pdf() in services/, which extracts text, chunks it, embeds each chunk with embedding-001, and stores the vectors in FAISS.
4. Respond. On success return 201 Created with a document_id, the chunks count, and "document ingested successfully."

# app/routes/upload.py  [+ illustrative, matches the walkthrough]
import logging
from fastapi import APIRouter, File, Form, UploadFile, HTTPException, status
from app.models import UploadResponse
from app.services.document_processor import DocumentProcessor

router = APIRouter()
log = logging.getLogger(__name__)
MAX_BYTES = 50 * 1024 ** 2          # 50 MB → bytes

@router.post("/upload", response_model=UploadResponse, status_code=status.HTTP_201_CREATED)
async def upload_document(
    file: UploadFile = File(...),
    description: str = Form(...),
):
    data = await file.read()
    if len(data) > MAX_BYTES:
        raise HTTPException(status.HTTP_413_REQUEST_ENTITY_TOO_LARGE, "File too large")
    if not data:
        raise HTTPException(status.HTTP_400_BAD_REQUEST, "Uploaded file is empty")

    result = await DocumentProcessor().ingest_pdf(data, description)
    return UploadResponse(
        message="document ingested successfully",
        document_id=result.id, chunks=result.n_chunks, description=description,
    )

From doorways to the workshop

Part 1 built the scaffold and one route. Part 2 fills in every remaining route, then the services where the real RAG work happens, and finishes with Docker packaging.

The remaining routes, and why they stay thin

Five more route files follow, repeating the discipline every time: a route receives the call, validates, delegates, and formats the reply, nothing more. The heavy work is imported from services/. Several service functions don't exist yet while the routes are written, so the imports show up "uncolored" in the editor; that's expected, they light up once the service code lands.

topics.py and suggestions.py, the LLM-as-helper routes

topics.py asks the LLM to read the uploaded document and return its 5–8 main topics, each with a one-sentence description. That list becomes the sidebar in the UI. The prompt is strict: "you are a document analyst… return exactly topic + one-sentence description, nothing else," so the output is easy to parse into a clean list.

suggestions.py generates exactly four starter questions a curious student would ask, the little prompt chips you see at the bottom of tools like Claude or ChatGPT before you've typed anything. It doesn't run the full RAG pipeline; it just feeds a sample of chunks to the LLM with a prompt: "generate exactly four specific, insightful questions, one per line, no numbering, no bullets."

query.py, the route that runs the full RAG pipeline

This is the endpoint behind the chat box: it takes a question and returns a grounded answer plus its sources. Unlike topics/suggestions, it runs the complete pipeline. The route itself stays thin, it validates, calls run_rag(...) from the service, and shapes the response, but the five conceptual stages are worth memorizing because they are "advanced RAG."

Why rewrite the query first? Users type vague or messy questions. Before searching, the LLM silently rewrites the question into a cleaner search query, the user never sees this, but it dramatically improves what FAISS retrieves. Then retrieval, cross-encoder re-ranking, and generation follow. The answer is returned with its source (document ID and page) so the user can verify where it came from.

evaluate.py and text_to_speech.py

evaluate.py scores an answer that was already generated, using the RAGAS framework, and logs the metrics. The route only defines the expected input/output shape and handles exceptions; the actual scoring lives in services/evaluator.py. It needs the four RAGAS inputs (question, answer, context, ground_truth) and returns metrics as JSON, also appended to a log file.

text_to_speech.py converts a text answer into spoken audio so the assistant can read replies aloud, with selectable voices (the default is "Nova"). It uses a GPT-4o audio model via OpenRouter, streams the response with httpx, extracts audio chunks, and wraps them in a WAV container for browser playback. A prompt tells the model to rewrite the text as a warm, spoken explanation and strip all markdown first.

The services, where the real work happens

This is "the most important part." Routes were doorways; services are the workshop. Five files: document_processor.py, embeddings.py, reranker.py, rag_pipeline.py, and evaluator.py, plus the empty __init__.py so they can be imported.

document_processor.py, the four-stage extraction cascade

Turns any PDF, however messy, into clean text, then chunks it, then builds the FAISS index. The hard part is "however messy": a PDF might be a clean digital file, a compressed corporate export, or a scanned paper document that's really just images. One extractor can't handle all of them.

Analogy, a set of increasingly specialized tools. You try the quick screwdriver first; if the screw's stripped, you reach for pliers; if that fails, the power tool. Each stage feeds the next, and a page only escalates to the slow, heavy tool if the lighter ones couldn't read it.

After extraction, the processor chunks the text (using the chunk_size and chunk_overlap from config.py), attaches metadata (page number, document ID, filename, description) to each chunk, embeds them, and writes the FAISS index to disk. It also exposes a load_vector_store() that reads that index back, raising "file not found" if it isn't there yet.

embeddings.py and reranker.py

embeddings.py wraps an embedding model so both documents and queries become vectors, with the same model on both sides. It defines two methods, embed_documents (for uploaded PDF chunks) and embed_query (for the user's question), and wraps both in retry logic (tenacity) so a flaky connection to the model server is retried instead of crashing the request.

reranker.py is stage two of retrieval: it scores each retrieved chunk against the query with a cross-encoder, sorts descending, and returns the top few. FAISS's bi-encoder hands it ~15 candidates; the cross-encoder reads each (query, chunk) pair together for a precise relevance score, sorts by that score, and keeps the best K (e.g. top 3–5). It logs how many it re-ranked and the top score for debugging.

# app/services/reranker.py  [+ illustrative, matches the walkthrough]
from sentence_transformers import CrossEncoder
from app.config import get_settings

settings = get_settings()

class Reranker:
    def __init__(self):
        self.model = CrossEncoder(settings.reranker_model)

    def rerank(self, query: str, docs: list, top_k: int = 5):
        pairs  = [(query, d.page_content) for d in docs]
        scores = self.model.predict(pairs)            # score each pair
        ranked = sorted(zip(scores, docs), reverse=True)  # high → low
        return [d for _, d in ranked[:top_k]]  # keep the best K

rag_pipeline.py, stitching it all into one chain

The orchestrator. It calls the other services in order to run a query end to end. Everything built so far, extraction, embeddings, retrieval, re-ranking, is glued together here into a single run_rag() function (the one the query.py route imports).

It holds two prompts, and keeping them straight is a common point of confusion:

The run_rag() execution order mirrors the five stages: optimize_query() → retrieve() (cosine similarity search in FAISS for top-K) → rerank() (cross-encoder) → build a formatted context from the surviving chunks and their metadata → generate the final answer with the RAG prompt. It returns the answer, the sources (with scores), and the rewritten query, the last of which feeds evaluation.

evaluator.py, scoring RAG quality with RAGAS

Grades a generated answer on four metrics and logs the scores. This closes the loop: you're no longer guessing whether the assistant is good, you're measuring it.

Each evaluation is written to a JSONL log (one JSON record per line) with a timestamp, a trimmed question, a ~200-character answer preview, and the metric scores. The preview is trimmed on purpose, evaluation is a developer-facing tool, so you don't need the full text, just enough to recognize the record alongside its scores.

Dockerfile and docker-compose, packaging the app

Part 1 explained why Docker exists; Part 2 writes the files. The Dockerfile is a recipe of commands that builds an image; docker-compose declares how the services run together.

# backend/Dockerfile  [+ illustrative, matches the walkthrough]
FROM python:3.11-slim              # small, fast base image from Docker Hub
WORKDIR /app
COPY requirements.txt .           # copy deps first (better layer caching)
RUN pip install -r requirements.txt
COPY . .                          # then copy the app code
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

# docker-compose.yml  [+ illustrative]
services:
  backend:
    build: ./backend
    ports: ["8000:8000"]      # API on :8000
  frontend:
    build: ./frontend
    ports: ["3000:3000"]      # UI on :3000

Half the practical value is watching real snags happen. Capture these, they're the mistakes you'd otherwise repeat.

Interview & practice Q&A

The real follow-up questions this material invites. Tap to reveal.

What is RAG, and how is it different from fine-tuning?

RAG (Retrieval-Augmented Generation) retrieves relevant text at question time and feeds it to the model as context, the model's weights never change. Fine-tuning actually retrains the model on new data, which is expensive and slow. RAG is preferred when your knowledge changes often or is document-specific, because you can add a new PDF instantly with no retraining.

Why use two-stage retrieval instead of a single vector search?

A bi-encoder vector search is fast but approximate, it embeds the query and chunks separately and compares. A cross-encoder reads the query and a chunk together, so it's far more accurate but too slow to run on every chunk. Two stages combine them: the bi-encoder recalls a wide set (top 15), then the cross-encoder re-ranks that small set for precision (top 5).

Why does every folder need an empty __init__.py?

It marks the folder as a Python package so its modules can be imported with dotted paths like from app.services.rag_pipeline import .... Without it, those imports fail. The file is intentionally empty, its mere presence is the signal.

What problem does Docker solve, and how does it differ from GitHub Actions?

Docker solves "works on my machine" by packaging the app with its exact dependencies into a container that runs identically anywhere. GitHub Actions is separate, it's CI/CD automation that runs steps (lint, test, build the Docker image, deploy) on every push. Docker is the box; Actions is the conveyor belt that builds and ships it.

What is CORS and why is the middleware needed?

CORS (Cross-Origin Resource Sharing) is a browser security rule that blocks a page from one origin calling a server at a different origin by default. Since the frontend runs on port 3000 and the backend on 8000, they're different origins. The CORS middleware makes the backend explicitly allow the frontend's origin, so the calls succeed.

Why is chunk overlap important?

Chunking splits a document into embeddable pieces, but a naïve split can cut a sentence or idea in half at the boundary. Overlap repeats the tail of one chunk at the start of the next, so the full context is preserved no matter which chunk gets retrieved. Typical overlap is ~10–15% of chunk size.

What are the RAGAS metrics and what inputs do they need?

RAGAS scores a RAG answer on faithfulness (is the answer grounded in the retrieved context?), answer relevancy (does it address the question?), context precision, and context recall. It needs four inputs: the question, the generated answer, the retrieved context, and a ground-truth reference answer.

How does upload.py enforce the 50 MB limit, and which status codes does it return?

It reads the uploaded bytes and compares their length against 50 × 1024². If larger, it raises HTTP 413 (Request Entity Too Large); if the file is empty, HTTP 400. On success it returns 201 Created along with a document_id, the chunk count, and a success message.

Where do secrets live, and why not commit them?

Real keys (OpenRouter, Google) go in a .env file that's listed in .gitignore so it never reaches GitHub. Only .env.example with blank placeholders is committed. A public repo is world-readable, so a committed key can be scraped and abused almost immediately.

Why route Gemini through OpenRouter instead of calling it directly?

OpenRouter is an aggregator that exposes many model providers behind one API key and URL. It lets you switch models by changing a config value instead of rewriting integration code, and centralizes billing and routing. The trade-off is an extra hop and dependency on the aggregator's uptime.

Why use a multi-stage PDF extraction pipeline instead of one library?

You can't predict what a user uploads, it might be a clean digital PDF, a compressed legacy file, or a scanned image with no text layer. Each engine (pdfplumber, PyPDF, PyMuPDF) is strong on different structures, and OCR handles image-only pages. The stages cascade so a page only escalates to slow OCR if the cheaper text engines fail on it.

Why rewrite the user's query before retrieval?

Users type vague or poorly phrased questions, which retrieve poor chunks. An LLM silently rewrites the question into a cleaner search query (the user never sees this), so FAISS returns more relevant results. It changes the phrasing, not the intent, purely to improve retrieval quality.

What does async/await buy you in the routes?

Async lets the server start a slow operation, like an LLM or network call, and do other work instead of blocking on it. Like a waiter serving other tables while one table decides, it raises throughput without extra threads. It's especially valuable here because nearly every route waits on model or network I/O.

How does the answer prompt prevent hallucination?

The RAG/answer prompt instructs the model to answer using only the provided context and to say honestly when the context doesn't contain the information, plus cite the source page. That grounding, combined with an explicit "admit when you don't know," stops the model from filling gaps with invented-but-plausible text. Returning sources also lets the user verify the answer.

Why can you run only three RAGAS metrics in production?

Faithfulness, answer relevancy, and context precision are reference-free, they only need the question, answer, and retrieved context. Context recall additionally requires a ground-truth answer to compare against, which you usually don't have live. So the code runs the three by default and conditionally appends context recall only when a ground truth is supplied.

Why does the Dockerfile copy requirements.txt before the source code?

Docker builds in cached layers. Since dependencies change far less often than application code, installing them in an earlier layer means editing a Python file doesn't invalidate the dependency-install step, so rebuilds skip reinstalling every library. It's a standard optimization that makes iterative builds much faster.