Session 6.1 · Advanced RAG & LLM Systems
From stuffing text into prompts to handing the model a map, then a first look at why hand-written prompts don't scale and what replaces them.
Section 1 · Copy this by hand
Telegraphic revision notes. Box the ⭐ / ⚠ / 💡 lines. Everything here is derived from the Detailed Notes below.
text-embedding-004 → 768 dims; OpenAI default → 1536UNWIND
(graph equiv. of SQL bulk UPDATE)LIMIT batchdb.create.setNodeVectorPropertydb.index.vector.queryNodes(idx, k=1, promptVec)MATCH path=(n)-[*1..2]-(m)
*1..2 = min 1 hop, max 2 hops, any edge type- not ->: traversal undirected (neighborhood, not direction)| Depth | Result |
|---|---|
| 1 | Too shallow, direct neighbors only |
| 2 | ✓ sweet spot, immediate + secondary context |
| 3 | Dangerous, expands exponentially |
| 4+ | Catastrophic, returns ~whole DB → crash |
Alpha -[ACQUIRED]-> Microsoft
WHERE size(neighbors) < 100
(blunt)MERGE dedupes by exact
string, not meaning → stray space = duplicate nodeSection 2 · Primary learning material
The problem that starts it all
You ask your assistant: "What's the name of the executive who leads the AI company that competes with the startup Microsoft just bought?"
Try to answer it with a plain vector search and you fail, because no single document says that sentence. Try to answer it with a pure database query and you fail too, because you don't know the exact names to look up. The answer only exists in the relationships between facts: Microsoft bought a startup → that startup has a competitor → that competitor has a CEO. That chain is exactly what Graph RAG is built to walk.
Graph RAG answers a question by handing the model a small map of how facts connect, instead of a pile of text and hoping the model connects them itself.
In ordinary retrieval-augmented generation (standard RAG), the strategy is simple and greedy: grab as many relevant text chunks as you can, cram them into the prompt, and let the model sort it out. It works for "what does this document say about X" but falls apart on questions whose answer is spread across several facts that were never written down together.
Graph RAG changes what you put in the context window. Instead of unstructured paragraphs, you feed the model the topology, the literal map of which things relate to which. Put another way: you build a "surgical theater" of a context window and tell the model, "here is a map of relationships, read the map and tell the user what you see."
The recurring stress-test question, "executive who leads the AI company that competes with the startup Microsoft just bought", is the whole course in one sentence. It needs a fuzzy entry point (vector), a relationship walk (graph), and a final read (LLM). Keep it in your head as the canonical multi-hop example.
Neo4j stores each node's embedding as a property attached directly to that node, then searches those embeddings with an approximate nearest-neighbor structure called HNSW.
Earlier, vectors lived in FAISS, a separate flat index held in memory, completely divorced from any graph. Neo4j does it differently: it attaches the numerical vector (that 768-dimensional float array) directly to the node entity, on disk. The node "Microsoft" carries both its label/properties and its embedding. The embedding captures the semantic meaning of the node's original text, e.g. Microsoft's text "a tech giant" holds far more searchable meaning than the bare token "Microsoft".
To find the closest vectors fast, Neo4j uses HNSW (Hierarchical Navigable Small World). You don't scan every node; you descend through layers of increasing detail.
Underneath every layer, "closeness" is measured with cosine similarity between the user's prompt vector and each node's embedding:
cosine(A, B) = (A · B) / (‖A‖ · ‖B‖)
Neo4j calculates this between the incoming query vector and node embeddings, and HNSW just decides which nodes are worth calculating it for.
You cannot just shove arrays into Neo4j and expect fast search, you must declare a formal vector index with a Cypher DDL statement. Three configuration rules matter:
| Rule | What to set | Note |
|---|---|---|
| Dimensions | Match the embedding model exactly | text-embedding-004 → 768; OpenAI default → 1536 |
| Similarity metric | cosine |
Same metric used at query time |
| Target | The embedding property on chosen labels |
e.g. Company, Executive, Document |
CREATE VECTOR INDEX alpha_fund_embeddings IF NOT EXISTS FOR (n:Company|Executive|Document) ON n.embedding OPTIONS { indexConfig: { `vector.dimensions`: 768, `vector.similarity_function`: 'cosine' }}
If your index dimensions don't match your embedding model, the database crashes on injection. A 768-dim index cannot accept a 1536-dim vector. Lock one embedding model per index and never mix. (This same "consistency" rule bites again in the Trinity build, see below.)
UNWINDTo embed thousands of nodes without melting your API quota, you pull a batch of un-embedded nodes, embed them all at once, and write them back in a single bulk operation.
The naive ("junior") approach loops one node at a time: MATCH … RETURN, call the embedding
API, write it back, repeat, 10,000 times. That takes hours and slams straight into API rate limits. The
production ("senior") approach is batch processing: pull the chunks, embed them in
bulk, push them back.
UNWIND unrolls a list of {id, vector} rows and writes them all
in one statement, the graph equivalent of a SQL bulk UPDATE.# fetch un-embedded nodes, batched MATCH (n) WHERE n.original_text IS NOT NULL AND n.embedding IS NULL RETURN id(n) AS id, n.original_text AS text LIMIT $batch # ...embed all texts in one API call, then bulk write: UNWIND $rows AS row MATCH (n) WHERE id(n) = row.id CALL db.create.setNodeVectorProperty(n, 'embedding', row.vector)
UNWIND is the key that makes embedding production-grade. Think of it as "the SQL bulk
UPDATE of Cypher": it takes a list parameter and unrolls it into rows so one query updates thousands
of nodes. Without it you're stuck in the per-node loop that hits rate limits.
Answer a multi-hop question by first finding where to start (vector), then walking the relationships (graph), then letting the model read the result (LLM).
Here is the core insight of the whole session: for a question like "the startup Microsoft just bought," neither tool works alone. Pure vector search (FAISS) can't follow relationships. Pure Cypher can't be written because you don't know the exact node names yet. So you go hybrid, in three phases.
Embed the user's vague prompt at runtime, then ask the vector index for the single closest node. This turns "the startup Microsoft just bought" into an exact database identity like Alpha AI, complete with a confidence score.
CALL db.index.vector.queryNodes('alpha_fund_embeddings', 1, $promptVector) YIELD node, score RETURN id(node) AS entryNode, labels(node), score
The user never wrote a WHERE name = … clause. You used spatial
mathematics (cosine distance) to convert a vague English phrase into the exact node
identity. That semantic entry point is what makes the rest of the traversal possible.
From the entry node you extract a localized subgraph, the ego graph: everything radiating outward from that "center of the universe" node, out to a fixed number of hops. You don't care about the whole city, only the entry node's neighborhood.
The tool for this is a variable-length path:
MATCH path = (start)-[*1..2]-(neighbor) WHERE id(start) = $entryNode RETURN path
*1..2 = follow between 1 and 2 hops, regardless of edge type.path returns the entire journey (node → edge → node → edge → node),
not just the destination.-[...]- have no arrows, the traversal is undirected.Leaving the relationship undirected is deliberate. For an ego graph you want the
neighborhood, not the direction, you don't care whether "Alpha acquired X" or "X acquired Alpha,"
only that they're connected. Traditional SQL would need one explicit JOIN per hop and breaks when
you don't know how many hops; *1..2 handles unknown depth in one clause.
The extracted subgraph is serialized into clean text and posed to the model alongside the user's question. The model reads the map and writes the final answer (covered in detail under Serialization).
Two hops is the empirically-chosen sweet spot: deep enough to capture indirect relationships, shallow enough not to drag in the whole database.
Why not just set five hops "to be safe"? Because path count explodes with depth in a connected graph. The trade-off, explicitly:
| Depth | What you get | Verdict |
|---|---|---|
| 1 hop | Direct neighbors only, misses the competitor's CEO | Too shallow |
| 2 hops | Immediate context + secondary consequences (parent companies, competitors' executives) | ✓ Sweet spot |
| 3 hops | Data volume expands exponentially | Dangerous |
| 4+ hops | In a connected corporate graph, effectively returns the whole DB → ~100k nodes into memory → server crash | Catastrophic |
Depth = 2 isn't a guess, it's the researched default. It captures "friend of a friend" (the
competitor's CEO) while staying bounded. Parameterize maxHops so it's tunable, but 2 is
the production answer for corporate knowledge graphs.
Flatten the retrieved paths into simple linear text so the model spends its reasoning on the answer, not on parsing brackets.
The model "has no eyes", it can't look at a graph visualization, and it shouldn't have to. If you dump raw Python dictionaries or JSON into the prompt, the LLM burns reasoning power parsing brackets instead of answering. Serialization is the translation layer: turn each path into a clean, human-readable line.
Alpha -[ACQUIRED]-> Microsoft, ready to cook with.
Serialization is "just formatting code", strip the brackets, join nodes and edges with arrows, but it materially improves answer quality. Clean input means the model's attention goes to reasoning, not to JSON archaeology.
Before serializing, trim the retrieved subgraph so a highly-connected node can't drown the model in tens of thousands of irrelevant paths.
The 2-hop traversal works beautifully for a niche node like Alpha. Now point it at Microsoft, a super node defined by its enormous degree (number of connected edges): thousands of employees, hundreds of acquisitions, tens of thousands of press releases. A 2-hop traversal from Microsoft can return ~50,000 paths. Stuff those in the prompt and you hit the token limit, dilute the model's attention, hallucinate, and burn API credits, all in one query.
The "junior" move is to shrug, "Gemini has a 1M-token window, stuff it all in." That's wrong. Two layers of pruning fix it:
Instruct Cypher to ignore over-generic, over-connected nodes by capping neighborhood size:
WHERE size(neighbors) < 100
This chops off the noisy hubs and keeps only reasonably specific relationships. It's crude but cheap.
This is the advanced technique: use vector math twice in the same query. Step 1 already used the prompt to find the entry node. Step 2 walks the paths and computes cosine similarity between the user's prompt vector and each neighbor's embedding. If a branch's meaning doesn't match (e.g. below a ~0.55–0.60 threshold), silently drop it.
# step 4 of a pruned query WITH neighbor, vector.similarity.cosine(neighbor.embedding, $promptVector) AS relevance WHERE relevance > 0.55 RETURN neighbor ORDER BY relevance DESC LIMIT $topK
Instead of 5,000 Microsoft connections, you return the ~5–10 paths that actually align with "competitor."
A neural network's attention degrades when the context is saturated with noise; facts buried in the middle of a huge prompt get ignored. Don't rely on a large context window to save you, prune before you send. And if the pruned graph doesn't contain the answer, train the model to say "I don't know" rather than hallucinate.
Wrapped in Python, a "surgically pruned" retriever guarantees the LLM receives at most ~10 curated, semantically-relevant paths no matter how massive the graph, and returns "no highly relevant relational paths found" when nothing clears the threshold.
A production-grade agent that routes each question to one of three engines, SQL, document vector search, or the live graph traversal, and lets the model chain them to answer.
The demo build (called Trinity Router because it routes to three tools) binds three functions to Gemini 2.5 Flash and runs an agentic loop. Two of the tools are mock stubs; the third is the real pruned graph retriever from above.
| Tool | Role | Status in demo |
|---|---|---|
query_financials |
Stub for a text-to-SQL engine (market-cap data) | Mock, returns canned data |
search_documents |
FAISS-style document vector search | Mock, returns canned data |
traverse_knowledge_graph |
Live: embed query → 2-hop pruned traversal → serialize | Real, hits Neo4j |
Because Gemini 2.5 Flash runs in thinking mode and may chain tool calls, e.g. run the
graph traversal, then a follow-up document search, then synthesize. The loop lets the model refine its
strategy mid-flight. It's capped at five rounds (MAX_TOOL_ROUNDS) so it
can't run forever.
On the canonical stress-test query, the router fires the graph traversal, walks from the Microsoft acquisition to the competitor to its CEO, and answers, in the demo, "Tim Cook leads Neural Net Labs, the AI company that competes with Alpha." [!] [+] Note: this is illustrative mock data from a toy graph, not a real-world fact; treat the names as placeholders for the mechanism.
During the demo, "Satya Nadella" appeared as a duplicate node. Cause: Cypher's MERGE
deduplicates only on exact string match, not meaning. An extra space, a slight
spelling variation, or a different surrounding context creates a separate node. Graph RAG
here has no LLM cleaning the data, so normalize your node identifiers before
merging or you'll get silent duplicates.
The demo used gemini-embedding-001 at 3072 dimensions (vs. 768 for
text-embedding-004). Higher dimensionality gives a sharper semantic entry point, but
remember the consistency rule: your Neo4j index must be configured for the exact same dimension
count.
Topic 2 · Declarative Optimization
The 2 a.m. scenario
Your production pipeline runs on a mega-prompt you spent weeks hand-tuning for Gemini. It's perfect. Then your CTO walks in: "We're switching the backend to Llama 3 to cut API costs." You swap the model, and the entire pipeline collapses, because every instruction was quietly overfit to one model's quirks. That fragility is the whole argument for DSPy.
Hand-writing prompts is unscientific tuning that's brittle, model-specific, and doesn't behave like real software engineering.
When you write a giant prompt telling the model exactly how to think, behave, and format, you're really hard-coding the system to one model's tokenization, biases, and attention mechanism. It works, until something changes. Three failure modes make manual prompting unscalable for production:
Don't confuse "the prompt works today" with "the prompt is robust." It's overfit to one model version, one document format, and one set of examples. Any of those three shifting silently degrades or breaks it.
DSPy is a compiler framework that treats your natural-language instructions and few-shot examples as tunable parameters, so an optimizer, not you, writes the prompts.
DSPy (Demonstrate, Search, Predict), created at Stanford, flips the paradigm. You stop hard-coding prompt strings. Instead you declare the pipeline logic (the architecture) and let an automated optimizer tune the instructions and few-shot examples as if they were mathematical weights.
if/else, "two pointy shapes on top = cat", brittle as glass, breaks when the
cat turns its head. Era 2 (feature engineering, 2000s): humans still hand-pick which features
(SIFT/HOG) matter. Era 3 (declarative, PyTorch/TensorFlow): you declare the layers, define a
loss, and an optimizer learns the billions of weights for you. DSPy is Era 3 for prompts.
In PyTorch you don't compute gradients by hand, you say "linear layer → conv layer," define a loss, and the optimizer sets the weights. DSPy does the same for language: you declare the pipeline and a metric ("is this answer correct?"), and the optimizer writes and tunes the prompts. The natural-language instructions become the tunable weights.
The lesson ended just as it introduced the first hands-on concept: the DSPy
signature, a declarative spec of a task's inputs and outputs (e.g.
question -> answer) that formalizes what a step does without you writing
how to prompt for it. [+] A full hands-on with signatures is
beyond the scope of this note.
The one-liner to remember: DSPy = programming, not prompting. "Demonstrate, Search, Predict", you demonstrate examples, it searches for good prompts/few-shots, and predicts. You own the logic; the compiler owns the wording.
Section 2 · Test yourself
Click each question, then say the answer aloud before expanding.
-[*1..2]-) because for
context-gathering we care that two nodes are connected in the neighborhood, not which direction the
relationship points.size(neighbors) < 100) as a blunt filter, then semantic edge
filtering (keep only neighbors whose embedding cosine-matches the query above a ~0.55 threshold) as
the scalpel.Alpha -[ACQUIRED]-> Microsoft, so the model spends its attention on answering, not
on parsing structure.MERGE sometimes create duplicate
nodes?MERGE deduplicates on exact string match, not meaning. A trailing space, a
spelling variant, or a differing property will produce a separate node, "Satya Nadella" duplicated
in the demo for exactly this reason. Normalize node identifiers before merging, since there's no LLM
cleaning the data mid-pipeline.UNWIND and remember MERGE dedupes on
exact strings, normalize identifiers or get silent duplicates.Build it · Chapter 07 · Atlas
The production move: put an embedding on every node, then run the three phases — vector jump (cosine to an entry node), traversal (the ego graph from Chapter 04), and pruning to tame super-nodes. embed is a real hashing bag-of-words vector — deterministic, no library — so you can read exactly why a node was chosen.
atlas/retrieve.py — vector jump → traverse → prune:
"""Chapter 07 — put a vector on every node, then retrieve in three phases.
Phase 1 vector jump : embed the question, cosine-match it to a node.
Phase 2 graph traversal : from that node, pull the ego graph (Chapter 04).
Phase 3 synthesis : hand the topology to a model (Chapter 08).
`embed` is a real (if tiny) hashing bag-of-words vector — no library, fully
deterministic. Swap it for a sentence-transformer and the pipeline is unchanged.
"""
import math
import re
from zlib import crc32
from atlas.graph import seed_graph
DIMS = 64
def embed(text):
"""A deterministic bag-of-words vector via feature hashing (crc32 is stable
across runs, unlike the built-in hash())."""
vec = [0.0] * DIMS
for word in re.findall(r"[a-z0-9]+", text.lower()):
vec[crc32(word.encode()) % DIMS] += 1.0
return vec
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
na, nb = math.hypot(*a), math.hypot(*b)
return dot / (na * nb) if na and nb else 0.0
def node_text(g, id):
"""What a node 'means': its name plus the relationships around it."""
rels = [f"{r} {o}" for r, o in g.out[id]] + [f"{r} {s}" for r, s in g.inc[id]]
return f"{id} {g.nodes[id]['label']} " + " ".join(rels)
def index(g):
"""Attach an embedding to every node — the vector rides on the node."""
for id in g.nodes:
g.nodes[id]["vec"] = embed(node_text(g, id))
return g
def vector_search(g, query, k=1):
"""Phase 1 — cosine-match the query to node vectors; return the best k."""
q = embed(query)
scored = sorted(g.nodes, key=lambda id: cosine(q, g.nodes[id]["vec"]),
reverse=True)
return scored[:k]
def prune(g, triples, query, cap=3):
"""Tame super-nodes: keep only the `cap` triples most relevant to the query."""
if len(triples) <= cap:
return triples
q = embed(query)
return sorted(triples, key=lambda t: cosine(q, embed(" ".join(t))),
reverse=True)[:cap]
def hybrid(g, query, hops=2, cap=6):
"""Vector jump -> ego traversal -> pruned topology, ready for synthesis."""
entry = vector_search(g, query, k=1)[0]
triples = g.ego(entry, hops)
triples = prune(g, triples, query, cap)
context = "\n".join(f"({s})-[{r}]->({o})" for s, r, o in triples)
return entry, context
if __name__ == "__main__":
g = index(seed_graph())
q = "the startup a big company acquired"
entry, context = hybrid(g, q, cap=4)
print(f"query: {q!r}\nentry node (vector jump): {entry}\n")
print(context)Run it — you'll see:
query: 'the startup a big company acquired' entry node (vector jump): Microsoft (Microsoft)-[acquired]->(Alpha) (Nimbus)-[competes_with]->(Alpha) (Q3 Report)-[mentions]->(Alpha)
This is one checkpoint of Atlas — a single project built across notes 04–14. Full code, setup, and the map of all checkpoints: Atlas — Build the notes ↗.