Multi-Agent Orchestration: Patterns and Pitfalls

Multi-agent orchestration is the practice of splitting a task across several LLM agents that communicate through structured handoffs, coordinated by a supervisor, a pipeline, or a shared workspace. Use it only when context isolation, independent verification, or true parallelism is the bottleneck — a single agent with well-designed tools is cheaper, faster, and easier to debug.

When you actually need multiple agents (and when one agent with tools wins)

Most teams do not need multiple agents. One agent with a competent tool loop handles the large majority of production workloads — triage, extraction, code edits, lookups. Splitting that into “researcher + planner + executor + critic” multiplies token spend and coordination bugs while leaving quality flat, because the bottleneck was never the agent count. It was the tool descriptions.

Multi-agent architecture buys you exactly four things:

• Context isolation. Sub-tasks need large, mutually incompatible context: a code agent needs the repository, a research agent needs dozens of pages. One window forces constant compaction and silent loss of detail.
• Independent verification. A critic sharing the producer’s context inherits its blind spots. Verification only works when the verifier arrives with different evidence or a different model family — a separate agent by construction.
• Genuine parallelism. Sub-tasks with no data dependency between them. Four independent lookups finish in the time of one. This is the only reason that lowers wall-clock latency instead of raising it.
• Privilege separation. Different tools, credentials, or model tiers per sub-task. The agent reading untrusted web content should not hold a database write credential.

The test that settles it: build the single-agent version first, score it on your eval set, then add the second agent and score again. If the score does not move, delete the agent. Almost nobody runs this test, because multi-agent designs are more fun to build than to measure.

Two structural facts make that test worth running. Token cost grows roughly linearly with agent count, since each agent pays for its own prompt, task, and retrieved context. Coordination failure surface grows faster than linearly, because every pair of agents is a potential handoff bug. Quality gains are sublinear and frequently zero. One agent is usually right when the task fits in one window and your tools stay in the low double digits with distinct names — the territory of our guide to AI agents and tool use. Reliability work — retries, validation, guardrails — pays off long before orchestration does, which is the subject of reliable AI agents. Orchestration amplifies the reliability you already have, including the absence of it.

Orchestration patterns

Choose by the shape of your dependency graph, not by which sounds most autonomous.

Pattern	Topology	Fits when	Breaks when	Relative token cost
Supervisor / router	One orchestrator decomposes and delegates; workers are stateless	Task mix is heterogeneous, routing is classifiable, and workers are reusable across products	The supervisor’s plan is wrong and no worker can push back — errors compound invisibly	~1 extra model call per step, plus the supervisor’s growing context
Pipeline	Fixed stages in a known order; each stage transforms an artifact	Stages are known at design time (extract → validate → enrich → render) and each is independently testable	An early stage’s error is undetectable until late; without schema checks at each boundary it propagates silently	Predictable: one call per stage, no coordination overhead
Debate / verifier	Producer, then critic, optionally a judge that arbitrates	Output is checkable (code with tests, math, cited claims) and errors are detectable but not preventable	Critic and producer share model family, context, or evidence — the critic just agrees	2–3× the single-agent cost; most expensive per unit of work
Blackboard	Agents read and write a shared artifact store and react to state changes	The workflow is emergent, long-running, and the step count is unknown up front	You cannot state a termination condition; also races, duplicate writes, and no owner of the final artifact	Unbounded unless you cap rounds and concurrency explicitly

The most common production shape is a supervisor over pipelines: the supervisor classifies and routes, each route is a fixed chain. You get routing flexibility without emergent behaviour, and each pipeline is testable in isolation with recorded inputs. Reach for the blackboard only when you genuinely cannot enumerate the steps.

Communication and handoff: the part that decides whether this works

Multi-agent systems fail at the seams, not inside the agents.

Message passing versus shared state. Default to message passing: each agent receives a bounded, purpose-built payload containing only what it needs. Isolation keeps contexts clean, makes every call replayable and cacheable, and lets you swap a worker without touching its neighbours. Use shared state only when artifacts are genuinely large and reused — a repository checkout, a dataframe — and even then pass references (IDs, paths, handles) rather than content. The worst of both worlds is the shared transcript: forwarding agent A’s entire history to agent B, which then inherits A’s wrong assumptions and A’s token bill. Forward conclusions, never the conversation.

Typed contracts, not prose. Every handoff should be a validated object with a schema in both directions. A handoff saying “summarize this well” is not a contract; it is a wish. This shape works in practice:

{
  "goal": "Return the three highest-impact API rate-limit mitigations",
  "constraints": ["cite a source per mitigation", "no vendor marketing claims"],
  "inputs": [{"kind": "doc_ref", "id": "s3://runs/8f2/limits.md", "version": 3}],
  "output_schema": {
    "status": "ok | partial | blocked",
    "items": [{"claim": "string", "source": "string", "confidence": "number"}],
    "blocked_on": ["string"]
  },
  "done_when": "items.length >= 3 AND every item has a non-empty source",
  "budget": {"max_turns": 4, "max_tokens": 12000},
  "idempotency_key": "run-8f2/step-2"
}

Five rules make handoffs unambiguous, each removing a specific class of bug:

• State the done-criterion as a mechanical assertion. items.length >= 3 can be checked by code. “A good summary” cannot, so it becomes an opinion.
• Return a structured envelope, never free text alone. Status ok, partial, or blocked, plus artifacts. The orchestrator branches on status; it never parses prose to learn whether a step succeeded.
• Give the worker a way to say “I can’t.” A blocked_on field listing missing information is the most effective loop-breaker in multi-agent design. An agent with no honest failure channel will guess instead.
• Carry invariants in every handoff. Global constraints are cheap to repeat and expensive to lose. Restating “never invent figures” in each payload prevents the drift where the third agent stops honouring a rule the first was given.
• Attach provenance. Every artifact carries producer ID and version, so you can bisect a bad output to the step that created it.

Cross-run persistence is a different problem from cross-agent handoff, and conflating them produces agents that confidently reuse stale context. Durable facts belong in a memory layer with its own write policy and eviction rules — see our guide to agent memory.

Failure modes

• Infinite loops and ping-pong. A asks B, B asks A, neither terminates. Detect: the same message hash recurring within a run. Fix: require monotonic progress — each round must yield a new artifact or shrink the open-questions count — and cap rounds near eight.
• Cost explosion. Re-passed transcripts, uncapped fan-out, retries that re-run the whole plan. Detect: tokens per completed task trending upward. Fix: a hard budget in code, a fan-out width cap, and fail-closed behaviour — abort with partial output rather than keep spending.
• Agents contradicting each other. Two workers return conflicting facts and the aggregator concatenates both. Detect: disagreements surfaced by a post-aggregation validation pass. Fix: one writer per artifact key, plus a precedence rule — verifier beats producer, newer evidence beats older, sourced beats unsourced.
• Lost context and amnesia. A downstream agent redoes finished work, or drops a constraint nobody restated. Detect: duplicated tool calls across spans. Fix: a run-scoped state object holding decisions and artifact references, included in every handoff.
• Cascading errors. A wrong fact from stage one poisons stages two through four, discovered at the end. Detect: output failures tracing back to one upstream span. Fix: validate at every boundary with schema checks plus a grounding check — does each claim resolve to an input artifact? Halt instead of propagating.
• Silent partial success. An agent returns confident prose that satisfies half the requirement. Detect: done-criterion assertions failing on otherwise “successful” steps. Fix: run those assertions as code after every handoff and treat a failure as a failed step, whatever the status field says.

One more failure is a security one rather than a quality one: privilege leakage. Scope tools and secrets per agent, not per system.

Cost and latency control

Multi-agent systems do not have a cost problem so much as a cost visibility problem: the orchestrator’s context grows with every step it observes, and that growth is the hidden quadratic.

• Cap turns per agent, in the loop. A max_turns value written in a prompt is a suggestion. A counter checked before each model call is a cap. When a worker hits its limit, treat it as a contract defect, not a worker failure.
• Budget the whole run, with a degrade threshold. At roughly 70% of budget, degrade: route remaining workers to a smaller tier, shrink fan-out, skip optional enrichment. At 100%, abort and return what you have. A budget that cannot stop a run is not a budget.
• Parallelize only independent steps. Derive the dependency graph first: if B consumes A’s artifact, it is serial however tempting concurrency looks. Where steps are genuinely independent, fan out — but cap concurrency so a burst does not become a wall of rate limits. Our AI API cost reduction guide covers the caching levers that compound with this.
• Keep the orchestrator’s state compact. Hold decisions, open questions, and artifact references — not transcripts. This is the largest single lever, because it changes the growth curve rather than the constant.

On latency, wall-clock is the critical path, not the sum. A verifier pattern adds two serial round-trips and is always slower than one agent; a fan-out of four independent lookups is faster. Reason in ratios rather than absolute prices, since those move: routing a worker from a frontier tier to a small/fast tier typically cuts that worker’s cost by roughly 5–10×, and putting the verifier on a mid tier while the producer stays on a frontier model usually captures most of the accuracy gain for a fraction of the spend. Per-worker model choice is itself a routing problem.

Observability across agents

Without a trace tree a multi-agent run is undebuggable, because the interesting failure is always relational — a bad handoff, a contradictory pair, a span that never returned. Emit one trace per run and one span per agent turn and tool call, with parent-child links so the tree reconstructs the delegation. Record on every span: run_id, agent_id, parent_span_id, turn index, model, tokens in and out, cost, latency, tool name, handoff schema version, retry count, status, and remaining budget.

Six metrics worth alerting on:

• Tokens per completed task — trending up means context bloat, usually transcript forwarding.
• Turns per agent, p95 — workers pinned at their cap mean the handoff contract is vague.
• Handoff schema-validation failure rate, by contract version — tells you which interface to fix, and whether your last prompt change made it worse.
• Contradiction rate — verifier rejections divided by total verifications; near zero means your verifier is agreeing, not checking.
• Fan-out width and concurrency, p95 — the leading indicator of a rate-limit storm.
• Runs hitting the budget ceiling — plus cost per run p95, so you catch the expensive tail, not the average.

Two implementation notes. Log the compact state object at each step and keep full transcripts in blob storage keyed by span ID — traces that store every prompt become unusable at exactly the scale where you need them. And instrument the orchestrator’s own calls as spans, since supervisors burn real tokens on planning and aggregation. For span schema, sampling, and correlating traces with evals, see our guide to LLM observability.

A minimal supervisor implementation

Below is a complete supervisor that routes to two worker agents and aggregates their output. It is deliberately small — no framework, one file — so you can see the four mechanisms that matter: a typed handoff, a structured envelope with an honest blocked status, a per-run token budget enforced in code, and a span per model call. The client speaks to any OpenAI-compatible endpoint, so pointing base_url at a relay such as qoraapi.com lets you move a worker between model families by changing one string.

"""
Minimal multi-agent supervisor: route -> delegate -> aggregate.
    pip install openai
"""
from __future__ import annotations
import json, time, uuid
from dataclasses import dataclass, field
from typing import Callable
from openai import OpenAI

client = OpenAI(base_url="https://qoraapi.com/v1", api_key="YOUR_KEY")

ROUTER_MODEL = "gpt-4o-mini"     # small/fast tier - the routing call is cheap
WORKER_MODEL = "gpt-4o"          # mid tier - workers do the real work
MAX_STEPS = 2                    # cap the plan, not just each agent
RUN_BUDGET_TOKENS = 40_000       # hard ceiling for the entire run

class BudgetExceeded(RuntimeError):
    """Raised when a run would exceed RUN_BUDGET_TOKENS."""

@dataclass
class Handoff:                   # the typed contract between agents
    goal: str
    constraints: list[str] = field(default_factory=list)
    inputs: list[dict] = field(default_factory=list)   # refs, never transcripts
    max_turns: int = 4

@dataclass
class Envelope:                  # every worker returns this shape, never free text
    status: str                  # "ok" | "partial" | "blocked"
    artifacts: list[dict] = field(default_factory=list)
    blocked_on: list[str] = field(default_factory=list)

@dataclass
class RunState:                  # compact state - NOT the conversation history
    run_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
    spent_tokens: int = 0
    artifacts: list[dict] = field(default_factory=list)
    decisions: list[str] = field(default_factory=list)
    spans: list[dict] = field(default_factory=list)

    def charge(self, tokens: int) -> None:
        self.spent_tokens += tokens
        if self.spent_tokens > RUN_BUDGET_TOKENS:
            raise BudgetExceeded(f"{self.run_id} spent {self.spent_tokens}")

def call(model: str, messages: list, state: RunState, agent: str) -> str:
    """One traced LLM call, with the attributes you will need at 3am."""
    t0 = time.time()
    resp = client.chat.completions.create(
        model=model, messages=messages, temperature=0,
        response_format={"type": "json_object"},
    )
    state.charge(resp.usage.total_tokens)
    state.spans.append({
        "run_id": state.run_id, "agent": agent, "model": model,
        "tokens": resp.usage.total_tokens,
        "latency_ms": int((time.time() - t0) * 1000),
        "finish": resp.choices[0].finish_reason,
    })
    return resp.choices[0].message.content

WORKERS: dict[str, Callable[[Handoff, RunState], Envelope]] = {}

def worker(name: str):
    def register(fn):
        WORKERS[name] = fn
        return fn
    return register

@worker("researcher")
def researcher(h: Handoff, state: RunState) -> Envelope:
    """Returns sourced facts only. No prose, no opinions."""
    raw = call(WORKER_MODEL, [
        {"role": "system", "content":
         "You are a researcher. Reply with JSON only. Every claim must carry a "
         "source. If you cannot source a claim, omit it."},
        {"role": "user", "content": json.dumps({
            "goal": h.goal,
            "constraints": h.constraints,
            "schema": {"facts": [{"claim": "string", "source": "string"}]},
        })},
    ], state, "researcher")
    return Envelope("ok", [{"kind": "facts", **json.loads(raw)}])

@worker("writer")
def writer(h: Handoff, state: RunState) -> Envelope:
    """Drafts from supplied facts. Says 'blocked' instead of inventing."""
    facts = next((a for a in h.inputs if a["kind"] == "facts"), {"facts": []})
    raw = call(WORKER_MODEL, [
        {"role": "system", "content":
         "You are a writer. Use ONLY the facts provided. If a required fact is "
         "missing, return status 'blocked' and list it in blocked_on."},
        {"role": "user", "content": json.dumps({
            "goal": h.goal, "constraints": h.constraints, "facts": facts["facts"],
            "schema": {"status": "ok|blocked", "draft": "string",
                       "blocked_on": ["string"]},
        })},
    ], state, "writer")
    out = json.loads(raw)
    if out.get("status") == "blocked":
        return Envelope("blocked", [], out.get("blocked_on", []))
    return Envelope("ok", [{"kind": "draft", "text": out.get("draft", "")}])

PLAN_SCHEMA = {"steps": [{"worker": "researcher|writer", "goal": "string",
                          "needs": ["artifact kinds"]}]}

def supervisor(task: str) -> RunState:
    state = RunState()
    plan_raw = call(ROUTER_MODEL, [
        {"role": "system", "content":
         "You are a supervisor. Decompose the task into the FEWEST steps. "
         f"Available workers: {list(WORKERS)}. Reply with JSON only."},
        {"role": "user", "content": json.dumps(
            {"task": task, "schema": PLAN_SCHEMA, "max_steps": MAX_STEPS})},
    ], state, "supervisor")

    for step in json.loads(plan_raw).get("steps", [])[:MAX_STEPS]:
        fn = WORKERS.get(step.get("worker", ""))
        if fn is None:
            state.decisions.append(f"skipped unknown worker {step.get('worker')!r}")
            continue
        needs = set(step.get("needs", []))
        inputs = [a for a in state.artifacts if a["kind"] in needs]
        try:
            env = fn(Handoff(goal=step["goal"], inputs=inputs), state)
        except BudgetExceeded as e:
            state.decisions.append(f"aborted: {e}")
            break
        state.artifacts.extend(env.artifacts)
        state.decisions.append(f"{step['worker']} -> {env.status}")
        if env.status == "blocked":
            # Halt, rather than let the next agent guess the missing input.
            state.decisions.append(f"halted, blocked_on={env.blocked_on}")
            break
    return state

def aggregate(state: RunState) -> str:
    draft = next((a["text"] for a in state.artifacts if a["kind"] == "draft"), None)
    if draft is not None:
        return draft
    return json.dumps(state.artifacts, indent=2)   # never fabricate on failure

if __name__ == "__main__":
    run = supervisor("Explain how API gateways absorb provider rate limits.")
    print(aggregate(run))
    print(json.dumps({"tokens": run.spent_tokens,
                      "decisions": run.decisions}, indent=2))

What it leaves out matters as much as what it includes. Each worker makes exactly one model call, so max_turns is declared but unused — the moment a worker runs its own tool loop, enforce that cap inside that loop, before each model call, never in the prompt. There is no retry logic, because blind retries on a failed handoff are how runs double their cost; retry transport errors and schema failures, never a semantic blocked.

To extend it, add a worker that takes kind: "draft" as input and returns a verdict — that is a verifier, and it needs no supervisor changes beyond one entry in WORKERS. That is the test of a good orchestration layer: adding an agent is a registration, not a rewrite.

Frequently asked questions

How many agents is too many?

Stop adding agents when the next one does not correspond to a distinct context, a distinct privilege boundary, or a distinct verifier. Systems that genuinely need multi-agent usually land between two and five. Beyond that, coordination cost dominates, and two agents sharing the same tools and context should be collapsed into one.

Should agents share a conversation history?

No. Share a compact run state — decisions, open questions, artifact references — and give each agent a purpose-built payload. Full-transcript sharing causes both context bloat and contamination: the downstream agent inherits an upstream agent’s wrong assumption along with the reasoning that produced it.

Do I need an orchestration framework?

Not to start. The supervisor above is about a hundred lines and covers routing, typed handoffs, budget enforcement, and tracing. Frameworks earn their dependency when you need durable execution across process restarts, checkpointed resumption mid-graph, or human-in-the-loop approvals — problems about state persistence, not about agents.

How do I stop agents from looping forever?

Three mechanisms together, not one. Cap total rounds per run and turns per agent in code. Require monotonic progress, so a round producing no new artifact terminates the run with partial output. And give every agent an honest blocked status with a blocked_on list — an agent with no way to report missing input will loop trying to invent it.

Conclusion

Multi-agent orchestration is a context-management technique, not an intelligence upgrade. Split when you need context isolation, independent verification, real parallelism, or privilege separation — and be able to name which one. Pick the topology from your dependency graph, make every handoff a validated contract with a mechanical done-criterion, enforce budgets and turn caps in code rather than prompts, and trace every agent turn so a bad handoff is visible instead of mysterious.

Start by building the single-agent version and measuring it. Then add one agent, re-measure, and keep it only if the score moved. To make the model-switching part a one-line change, put an OpenAI-compatible gateway in front — the supervisor above runs unchanged against many models through one endpoint.

Related reading

• Building Reliable AI Agents: Guardrails, Retries, and Human-in-the-Loop
• AI Agents 101: Orchestrating Multi-Step Tasks with Tool Use
• Giving AI Agents Memory: Working, Episodic, and Retrieval Memory
• LLM Observability: Monitoring AI API Usage, Latency and Cost