Persistent Agents & Context Management

Section 1Notebook Summary

Hand-copyable revision notes. Every line here is derivable from the Detailed Notes below.

Section 2Detailed Study Notes

A customer opens a chat on Monday about a double charge. They come back Thursday to finish it. If your agent forgot everything the instant the server restarted, they have to explain the whole thing again — and they will not be gracious about it. So you give the agent memory: every message, every tool result, written to disk. It works. Then, fifty turns into a knotty case, that same agent starts quoting the wrong invoice number, and the bill for a single reply quietly costs a hundred times what the first one did. Memory solved one problem and created its mirror image. This lesson is about handling both ends.

2.1Why an agent needs memory at all

Persistence is giving the agent a place to write down what happened so it isn't lost when the program stops. A plain chat agent holds the conversation in RAM. Kill the process — a deploy, a crash, an autoscaler reclaiming the box — and the entire conversation evaporates. For a throwaway demo that's fine. For a support product where one case spans days, it's fatal.

The mechanism is the checkpoint: after each step of the graph, the agent's full state is serialized and saved. A checkpointer is the component that does the saving. In the last session the state lived only in memory (a MemorySaver); here we swap in one backed by a real database so it outlives the process.

2.2Time travel: state history and rewind

Because every step is checkpointed, the whole conversation is a list of saved states you can walk back through — that's "time travel." The naive read is that this is a debugging convenience. The sharper read, and the one the instructor stressed, is that in regulated industries a system that makes consequential decisions without a history of those decisions is non-compliant by design. The audit trail isn't a nicety; it's the point.

The practical payoff is rewind: correcting a mistake surgically instead of restarting the conversation.

The worked example: a pricing tool returns a wrong price because of a data bug (the instructor's real-world echo — a ₹25,000 item briefly listed at ₹250). The agent quotes the bad number to the customer. Once the service bug is fixed, you don't apologize and restart. You reload the checkpoint just before the pricing call, replace the bad tool output with the right one, and resume. Everything before that point — already correct — is reused as-is.

2.3Human-in-the-loop: five things time travel unlocks

Once you can pause, inspect, fork, and restore states, a human can be inserted anywhere in the loop. "Human-in-the-loop" (HITL) just means a person reviews or corrects the agent at some point. Time travel is the infrastructure that makes all five of these cheap:

2.4Interrupt and resume mechanics

LangGraph has a formal pause button: compile the graph with interrupt_before (or interrupt_after) a node, and execution stops there and hands control back. The current state is exposed for inspection and editing. When the reviewer is done, you call invoke again with the same thread_id; the state updates and the graph resumes from where it paused.

The reason this is robust: the pause is a checkpoint with a "waiting for human" flag. If the server restarts while the graph is parked, the pause is still on disk — the human can return hours later, load the state, make changes, and continue. Compare that to traditional HITL, which either keeps the process alive for the whole review window (expensive and fragile) or bolts on a custom queue to route the request out to a human and the answer back. LangGraph collapses that complexity into the checkpoint infrastructure you already have.

2.5Hands-on: the persistent support agent

The entire graph stays the same — the only production change is where the state is saved. The graph is still classifier → agent ⇄ tool → respond with conditional edges between them. Two edits promote it to production-grade: a database-backed checkpointer at compile time, and a thread configuration at invoke time.

from langgraph.checkpoint.sqlite import SqliteSaver

# One local file holds every thread's checkpoints
checkpointer = SqliteSaver.from_conn_string("support.db")

# Same builder as before — only .compile() changes
graph = builder.compile(checkpointer=checkpointer)

# thread_id is what isolates one conversation from another
config = {"configurable": {"thread_id": "u4-s00001"}}
graph.invoke({"messages": [user_msg]}, config)

[+] The code is idiomatic LangGraph shown for precision; the lesson described these APIs (SqliteSaver, thread_id config, get_state_history, update_state) without walking every line.

The behaviour that makes it click: send a second message on the same thread and the agent already has the full prior context loaded from disk. Ask "what was the outstanding balance?" and it answers 998 directly — the node path shrinks to classify → agent → respond, with no tool call, because the CRM result from the first turn is already sitting in the checkpoint.

# Full history is restored from support.db for this thread
config = {"configurable": {"thread_id": "u4-s00001"}}

graph.invoke(
    {"messages": [HumanMessage("what was the outstanding balance?")]},
    config,
)
# -> "Your outstanding balance is 998."
# classify -> agent -> respond   (get_customer_details is NOT called again)

2.6The time bomb: memory that only grows

The persistent agent faithfully keeps everything — and "everything" is the problem. Every tool call appends a tool message; every agent turn appends an AI message; the checkpointer serializes all of it. There's no forgetting. The architecture that fixed no-memory now grows without bound, and it detonates in production without warning.

A three-step react loop per turn (agent thinks → tool runs → agent responds) means each turn adds roughly three messages. After 10 turns that's ~30 messages; after 50 turns, ~150. A genuinely complex case spanning several days easily reaches 150+ messages — and the full list is sent to the model on every single inference call.

Figures are the lesson's worked estimates (≈200 tokens/message, 3 messages/turn), for scale — not measured benchmarks.

Three failure modes ride along together: cost grows linearly, reasoning quality degrades as the context gets noisier, and eventually the history exceeds the model's window and the API errors out — the production agent crashes mid-conversation, on a live customer. Last session solved the no-memory problem; this session solves the too-much-memory problem.

2.7Why a bigger window is not the fix

"Just use a million-token window" is an architectural error, not a solution. A larger window is more expensive and, more importantly, doesn't make reasoning better. Two mechanisms explain why.

Attention dilution

A transformer spreads a fixed attention budget across all token pairs. As the sequence grows, the share landing on any one token shrinks. A critical fact stated at turn 2 of a 50-turn case is competing with 48 turns of subsequent noise for the model's attention.

The needle-in-a-haystack (a.k.a. "lost in the middle") benchmark makes it concrete: hide one relevant fact at varying positions in a long document and ask the model to find it. Recall that sits around 90–95% at tens of thousands of tokens can fall to roughly 60–70% at 100,000 tokens, worst when the target is buried in the middle. Adding tokens doesn't improve retrieval — it makes it worse.

Reasoning chain degradation

Chain-of-thought reasoning is sensitive to how dense and close the relevant context is. Each hop that slightly mis-weights or mis-remembers an earlier fact compounds; over a long noisy context these errors accrete into answers that are internally consistent with the mess but wrong about reality.

2.8The three-layer framework

The architect's job is to write down policies for how information flows through the window over time — and those are domain decisions, not something to hand the model. Every piece of history is sorted into one of three layers:

Verbatim means exactly as said — for the last few turns, exact wording (account IDs, amounts, promises) matters. Semantic compression keeps the meaning of older content while throwing away the phrasing. Discard removes procedural noise that carries no bearing on future reasoning. These judgments are architectural: they depend on your domain, and they must be explicit in code.

2.9Pruning: the sliding window

The simplest strategy is to keep only the last N messages and drop the rest. N is the one design knob. A small window (4–6) slashes cost but creates a hard memory horizon: anything older is permanently gone, and the agent doesn't even know it existed. A large window (20–30) preserves more but saves less — and no value of N escapes the core limitation: once a message falls outside the window, it's irretrievable.

So is pruning ever right? For an FAQ chatbot where each question is independent, yes — nothing before this question should influence the answer. For a support agent tracking one evolving case, absolutely not. That's the domain specificity of the choice.

2.10Summarization: lossy compression done on purpose

Instead of discarding old messages, compress them. When the message count crosses a threshold, a dedicated LLM call produces a dense summary of the older messages, stored in its own state field and injected into the agent's system prompt.

You are a precise summarizer for customer support.
Preserve EXACT values (account IDs, invoice numbers, amounts).
Compress; do NOT keep verbatim wording.
Delete acknowledgements and procedural back-and-forth.
Output ONE paragraph, max 5 sentences. No preamble, no labels.

2.11The summarization node & separation of concerns

Knowing how to summarize isn't enough — where the logic lives in the graph decides whether you can maintain it. The agent node has one job: pure reasoning (invoke the LLM with the bound tools and current messages). Stuff summarization inside it and you've given one node two jobs with different performance profiles, failure modes, and testing needs.

The fix is separation of concerns: a dedicated summarization node with one job — take the message list, compress the older ones into a summary, update state, return. Well-defined input and output means it can be unit-tested (assert the summary still contains the required facts) and swapped (upgrade flash → pro) without touching — or re-testing — the critical agent node and its whole react loop.

def summarize_node(state):
    msgs = state["messages"]
    old, recent = msgs[:-VERBATIM], msgs[-VERBATIM:]     # layer 2 vs layer 1
    summary = summarizer_llm.invoke(SUMMARY_PROMPT + render(old))
    return {"system_summary": summary.content, "messages": recent}

2.12Conditional routing: the maintenance loop

The summarization node must not run on every turn — that would erase the savings and add latency to every response. It fires only when the message count exceeds a threshold (a tunable parameter, kept outside the production DB so it can be changed freely). Routing is a LangGraph conditional edge, the same pattern used everywhere else, applied to a new decision.

def route_after_agent(state):
    if len(state["messages"]) > THRESHOLD:      # too big -> maintain
        return "summarize"
    if state["messages"][-1].tool_calls:        # needs a tool -> tools
        return "tools"
    return END                                  # done

Both the summarize node and the tool node always loop back to the agent node. Summarization is a maintenance operation, not a terminal state — which is exactly why it's called a maintenance node. The shape is the react loop with one extra branch, nothing more exotic.

[!] The lesson quoted the react-loop circuit breaker as both 12 (the MAX_ITERATIONS constant) and "maximum 5" in passing; treat 12 as the configured cap and the smaller number as a typical per-turn count. Also: 1M-token windows were attributed once to Gemini 1.5 Pro and once to Gemini 2.5 Flash — the exact model label isn't load-bearing for the point.

2.13Interview & practice Q&A

2.14Key takeaways

Section 3Microexercises

Reading proves you recognise the idea; these prove you can use it. Attempt each before opening the walkthrough.

3.1Build it — the routing edge

THRESHOLD = 12
# state["messages"]      -> list of message objects
# state["messages"][-1]  -> last message; .tool_calls is truthy when a tool is requested
# valid returns: "summarize" | "tools" | END

def route_after_agent(state):
    ...   # your code

from langgraph.graph import END

def route_after_agent(state):
    if len(state["messages"]) > THRESHOLD:
        return "summarize"
    if state["messages"][-1].tool_calls:
        return "tools"
    return END

3.2Diagnose it — the entangled node

def agent_node(state):
    # summarize the history, then reason
    summary = summarizer_llm.invoke(SUMMARY_PROMPT + render(state["messages"]))
    state["system_summary"] = summary.content
    msgs = [SystemMessage(state["system_summary"])] + state["messages"]
    return {"messages": [llm_with_tools.invoke(msgs)]}

def agent_node(state):                     # reasoning ONLY
    system = [SystemMessage(state.get("system_summary", ""))]
    msgs = system + state["messages"]
    return {"messages": [llm_with_tools.invoke(msgs)]}

def summarize_node(state):                 # maintenance ONLY
    old, recent = state["messages"][:-VERBATIM], state["messages"][-VERBATIM:]
    summary = summarizer_llm.invoke(SUMMARY_PROMPT + render(old))
    return {"system_summary": summary.content, "messages": recent}

# route_after_agent (from 3.1) fires summarize_node only when count > THRESHOLD