Cadre

Production Scaling — Queues, Checkpoints, Durability

Scaling multi-agent systems to thousands of concurrent runs requires **durable execution**. LangGraph's runtime writes a checkpoint after each super-step keyed by `thread_id` (Postgres by default); worker crashes release a lease and another worker resumes. Agents can sleep indefinitely waiting for human input. **MegaAgent** (arXiv:2408.09955) ran a per-agent producer-consumer queue with three states (Idle / Processing / Response) and two-layer coordination (intra-group chat + inter-group admin chat). **Fiber/async** beats thread-per-job for LLM streaming: threads sit idle 99% of the time waiting for tokens, fibers cooperatively yield on I/O. Counterpoint: Ashpreet Bedi's "Scaling Agentic Software" argues for **FastAPI + Postgres + nothing else** until load proves otherwise — simple architectures go further than expected. This lesson builds a durable checkpoint log, a per-agent work queue with state transitions, an async-vs-thread demo, and lands the pragmatic "start simple" rule.

Learn + Build 1.3 hr Python (stdlib, `asyncio`, `sqlite3`)

Prerequisites: Phase 16 · 09 (Parallel Swarm Networks), Phase 16 · 13 (Shared Memory)

#Problem

A prototype multi-agent system works on one laptop with three agents in an in-memory event loop. You move to production:

Agents sometimes run for hours (long research, human-in-the-loop waits).
Worker processes crash. Restarting loses state.
Peak load is 10x average; you need horizontal scaling.
Users pay per agent-run; you need exactly-once semantics for charging.

The in-memory event loop does none of these. You need a durable execution layer underneath. The 2026 canonical options are:

A workflow engine with checkpoints (Temporal, LangGraph runtime).
A message queue with a state store (Postgres + SQS/RabbitMQ).
Actor-model frameworks (MegaAgent's producer-consumer per agent).
Hand-rolled FastAPI + Postgres (Bedi's argument).

This lesson builds a miniature of each.

#Problem

Keep going. The rest of this lesson is one step away.