The Supervisor Orchestrator

Notebook Summary

Hand-copy target · fully derivable from the Detailed Notes below.

Detailed Study Notes

A customer sends one message: "My API has been returning 500 errors since yesterday. Also, I just noticed a duplicate charge on my account. I need both fixed urgently." Your routing code reads the ticket, decides it is technical, and hands it to the tech-support agent. Tech support fixes the API and closes the ticket. The duplicate charge is never investigated. The customer escalates, angry, and now you are paying for a human agent to clean up what the machine skipped. Nothing crashed. The code did exactly what it was told. That is the failure this whole lesson is built to prevent.

2.1Why static routing breaks

Static routing is a lookup: read one field, return one path. It is fast, free, and perfectly reliable — for tickets that have exactly one intent.

The routing you have built so far (call it route_by_category) reads a single field like category and returns a single string. It was designed for clean, single-intent tickets: "Refund my payment." That covers maybe 70% of real traffic. The remaining traffic is where it quietly fails: multi-intent tickets (bill and API), ambiguity (is "account suspended" a billing problem or a fraud problem?), and conversations whose intent shifts halfway through.

2.2What a supervisor actually is

A supervisor is a dedicated LLM node that reads the shared state and decides which worker runs next. Not a router, not an agent — a coordinator with exactly one job.

The distinction matters. A router reads one field and returns one string. An agent does work — calls tools, writes responses. The supervisor sits above both: it reasons across the whole ticket history and every specialist's findings, then names the next step. Four properties define it:

2.3The single-endpoint requirement

Customers must always talk to exactly one door, no matter how many specialists live behind it.

A retail bank with ten million customers exposes one chat window. Those customers must never know whether they are hitting a triage API, a billing API, or a fraud API. This is not a convenience — it is an architectural guarantee. When a direct hand-off wires triage straight to tech support with a hard-coded edge, the caller becomes coupled to the internals: add a specialist and you must edit the pipeline, rewire conditional edges, retest every path, and renotify every client. Edge count grows super-linearly — a combinatorial explosion.

Route everything through one supervisor and you get the Open-Closed Principle applied to a multi-agent system: open for extension (register a new subgraph in the master graph) and closed for modification (the external API contract, the client code, and the other subgraphs all stay untouched). You grow from 2 agents to 20 with zero client updates, zero contract changes, zero retesting of unchanged graphs.

2.4Two routing strategies: deterministic vs. semantic

Deterministic routing is plain Python on a field you already trust; semantic routing is an LLM interpreting raw text. They cost wildly different amounts — choose by what the decision actually needs.

Deterministic (Python) routing uses if/else, match, or dict dispatch on a state field whose value is already validated and classified. Once triage has written ticket_type = "technical" to shared state, routing on that field costs $0, runs in under a millisecond, and is 100% reliable — not 99.7%, because if statements do not hallucinate. It is unit-testable with a plain assertion.

Semantic (LLM) routing is for raw, unstructured, ambiguous, or multi-intent input where no prior classification exists. This is the one place the LLM earns its keep — and it pays a latency tax: 200–500ms (up to ~2s) of wall-clock time per call, plus token cost, even for a fast model.

Watch the compounding: a request making five LLM routing decisions at ~1s each makes the user wait five seconds of dead time before seeing anything — five API calls paid for, zero user-visible value if it fails. The mechanical alternative would decide the same thing in under a millisecond at zero cost.

2.5Structured output: Pydantic Literal

The supervisor must return a machine-checkable value, never free text. A Literal type pins the output to a closed set of valid worker names.

Picture the supervisor replying "I think billing should handle this." The graph's conditional edge tries to read next_worker, finds a sentence instead of a node name, and raises a KeyError — at 2am, in production, leaving the request permanently unserviced. The fix is to constrain the output space before the model ever speaks.

from typing import Literal
from pydantic import BaseModel

class SupervisorDecision(BaseModel):
    # next_worker can ONLY be one of these exact strings.
    next_worker: Literal["triage", "tech_support", "fraud_handler", "FINISH"]

Bind it with llm.with_structured_output(SupervisorDecision). LangChain converts the Pydantic schema to a JSON schema, hands it to the LLM's structured-output feature, and parses the response back into a typed object. The constraint is enforced at two points: first by the LLM API (it is steered to emit only approved values), then by Pydantic on deserialization (any stray value raises a validation error). The calling code receives a validated object with a known attribute type — not a raw string it must parse by hand.

2.6The Supervisor Purity Principle

The supervisor has one job and must do only that job. The moment it starts doing business logic, it becomes untestable.

2.7The FINISH condition

The supervisor decides when the job is done — not the workers. FINISH is a routing decision like any other, one the supervisor has to earn.

A worker that signals END on its own bypasses oversight: the supervisor never gets to check whether the billing dispute or fraud signal still sitting in messages was handled. Instead, every worker returns to the hub, and the supervisor runs three checks before it terminates:

This is what prevents premature termination on complex, multi-intent tickets — the exact failure static routing produces silently. FINISH earned by synthesis is the difference between "API fixed, ticket closed" and "API fixed, billing addressed, then closed."

2.8Loop safety: delegation_count

A supervisor with no stop condition can loop forever. delegation_count is to the supervisor what an iteration limit is to an agent loop.

The loop is easy to fall into: the supervisor routes to tech, tech returns to the hub, the supervisor doesn't register progress and routes to tech again — token hemorrhage on circular logic. A counter is the circuit breaker. Each delegation increments it; once it hits the max, the supervisor forces FINISH at zero extra cost and delivers whatever response was last generated.

2.9Designing the supervisor prompt

The prompt is the supervisor's only source of intelligence. It has three load-bearing parts: an explicit prohibition, an enumerated worker list, and a compact state summary.

The prohibition must be written with maximum specificity — "only," "never," "under any circumstances" — because a completion machine will otherwise try to be helpful. The approved list enumerates the exact valid strings for next_worker: it shrinks the output space, removes the need to invent a key from scratch, and gives the Pydantic Literal a target to validate against. The state summary passes a concise, structured digest instead of the full conversation — critical because the supervisor is called many times per request, so token savings multiply.

SUPERVISOR_PROMPT = """You are a support supervisor orchestrator.
Your ONLY function is to select the next worker from the approved list.
You must NEVER answer the user directly, NEVER call tools, and NEVER
generate content intended for the user, under any circumstances.

Approved workers: triage, tech_support, fraud_handler, FINISH

Decision rules:
  1. If the final response addresses every issue: return FINISH
  2. If a billing or technical issue is still unaddressed: return tech_support
  3. If suspicious activity is detected: return fraud_handler
  4. If delegation_count > 3: return FINISH
  Default to FINISH when in doubt.

Return only: {"next_worker": "<one approved value>"}"""

2.10Building the node: seven stages

The supervisor node runs a fixed seven-stage cycle, with the LLM-touching stages wrapped so a failure degrades to a safe route instead of a crash.

from langgraph.graph import END
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.messages import SystemMessage, HumanMessage

MAX_DELEGATIONS = 10  # hard circuit breaker

def supervisor_node(state: SharedState) -> dict:
    # 1 - extract the fields the routing decision needs
    count = state.get("delegation_count", 0)

    # 2 - enforce the safety limit BEFORE any LLM call
    if count >= MAX_DELEGATIONS:
        return {"next_worker": "FINISH", "delegation_count": count}

    next_worker = "triage"  # safe default if the LLM path fails
    try:
        # 3 - fast, cheap, deterministic model
        llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", temperature=0)
        router = llm.with_structured_output(SupervisorDecision)

        # 4 - prompt = system prohibition + compact state summary
        messages = [
            SystemMessage(content=SUPERVISOR_PROMPT),
            HumanMessage(content=summarize_state(state)),
        ]

        # 5 - invoke; receive a VALIDATED SupervisorDecision object
        decision = router.invoke(messages)
        next_worker = decision.next_worker
    except Exception:
        next_worker = "triage"  # timeout / rate limit / bad schema

    # 6 + 7 - increment the count and return the state update
    return {"next_worker": next_worker, "delegation_count": count + 1}


def supervisor_router(state: SharedState) -> str:
    # PURE Python. No LLM. Reads the field the node already validated.
    worker = state["next_worker"]
    return END if worker == "FINISH" else worker

builder.add_node("supervisor", supervisor_node)
builder.add_node("triage", triage_subgraph)
builder.add_node("tech_support", tech_support_subgraph)
builder.add_node("fraud_handler", fraud_subgraph)

builder.set_entry_point("supervisor")

# supervisor -> (worker | END), chosen by the pure router
builder.add_conditional_edges(
    "supervisor",
    supervisor_router,
    {
        "triage": "triage",
        "tech_support": "tech_support",
        "fraud_handler": "fraud_handler",
        END: END,
    },
)

# THE HUB INVARIANT: every worker returns to the supervisor
for worker in ("triage", "tech_support", "fraud_handler"):
    builder.add_edge(worker, "supervisor")

2.11Auditability: one hub, total visibility

Because every transition passes through the supervisor, one node sees 100% of the execution flow. Instrument the hub and you get a complete, queryable audit trail for free.

Each decision records which worker was selected, which state was observed at selection time, the delegation count, and how long the LLM call took. That answers audit questions like "why did this complaint go to billing instead of tech?" down to the exact state. In a direct-handoff topology this is impossible — a mid-pipeline failure in one subgraph is invisible to every other node, and the user just gets no response. With a hub, that failure is visible: log it, increment an error counter, retry or escalate.

2.12Interview & practice Q&A

2.13Key takeaways

Microexercises

Reading proves you recognize the pattern. These prove you can build it. Attempt each before opening the walkthrough.

3.1Build it — the decision model and pure router

from typing import Literal
from pydantic import BaseModel
from langgraph.graph import END

class SupervisorDecision(BaseModel):
    next_worker: Literal["triage", "tech_support", "fraud_handler", "FINISH"]

def supervisor_router(state: dict) -> str:
    worker = state["next_worker"]
    return END if worker == "FINISH" else worker

# checks
assert supervisor_router({"next_worker": "tech_support"}) == "tech_support"
assert supervisor_router({"next_worker": "FINISH"}) == END

3.2Diagnose it — the supervisor that never stops

MAX_DELEGATIONS = 10

def supervisor_node(state: SharedState) -> dict:
    count = state.get("delegation_count", 0)
    if count >= MAX_DELEGATIONS:
        return {"next_worker": "FINISH", "delegation_count": count}

    llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", temperature=0)
    router = llm.with_structured_output(SupervisorDecision)
    decision = router.invoke([
        SystemMessage(content=SUPERVISOR_PROMPT),
        HumanMessage(content=summarize_state(state)),
    ])
    # returns the routing decision
    return {"next_worker": decision.next_worker}

    # was: return {"next_worker": decision.next_worker}
    return {
        "next_worker": decision.next_worker,
        "delegation_count": count + 1,   # arm the circuit breaker
    }