Your Multi-Agent System Is a House of Cards: Why You Need a Circuit Breaker, Not Just a Retry
I’ve seen it happen more times than I care to count. A team builds a beautiful multi-agent system. Agents talk to each other. Workflows flow. Everyone’s happy.
Then one API goes down.
Startup Software Development Case Study: How to Cut Costs by 60% and Ship Faster
Summary: A Vietnamese startup needs fast, affordable software without sacrificing quality. This startup software development case study shares… ...
Suddenly, every agent in the chain is hanging. Timeouts pile up. Memory blows. The whole system locks up tighter than a production database during a full table scan.
And what’s the first thing most developers reach for? A retry loop.
Why Smart CTOs Hire Vietnamese Developers: A Data-Driven Guide for 2025
TL;DR: Vietnam offers the best value in offshore development today—strong technical universities, 95% developer retention, 40% cost savings… ...
That’s not a fix. That’s a prayer.
Let me show you why retries are the wrong tool for multi-agent orchestration, and how the circuit breaker pattern actually saves your system from itself.
The Retry Fallacy
Here’s the problem with retries in a multi-agent system: they assume the failure is transient. But what if it’s not?
- What if the downstream LLM API is rate-limiting you?
- What if the vector database is re-indexing?
- What if the external service is just… dead?
A retry loop in these scenarios doesn’t just waste time. It actively makes things worse. Each retry consumes memory, holds onto connections, and blocks other agents from doing useful work.
I worked with a client in Ho Chi Minh City last year who had a 12-agent pipeline for document processing. One of their agents called a third-party OCR service. That service went down for 3 minutes.
The retry logic in their orchestrator spawned 47 concurrent retries before the timeout kicked in. The system OOM’d in 90 seconds.
Three minutes of downtime turned into 45 minutes of recovery.
The Circuit Breaker: Your System’s Immune System
The circuit breaker pattern is dead simple. It has three states:
- Closed: Everything’s fine. Requests flow through.
- Open: Something’s broken. Requests fail fast without even trying.
- Half-Open: Testing the waters. Let a single request through to see if the service recovered.
That’s it. But the implementation details matter. A lot.
The Naive Implementation (Don’t Do This)
python
class NaiveCircuitBreaker:
def __init__(self, failure_threshold=5):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.state = "CLOSED"
def call(self, func):
if self.state == "OPEN":
raise Exception("Circuit is open")
try:
result = func()
self.failure_count = 0
return result
except Exception:
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.state = "OPEN"
raiseThis works for a toy example. In production, it’s dangerous. Why? No time-based recovery. Once it’s open, it stays open forever. You’d need a manual reset.
The Production-Ready Version
Here’s what we actually run at ECOA AI for our agent orchestration platform:
python
import time
import asyncio
from enum import Enum
from dataclasses import dataclass
from typing import Callable, Optional
class CircuitState(Enum):
CLOSED = "CLOSED"
OPEN = "OPEN"
HALF_OPEN = "HALF_OPEN"
@dataclass
class CircuitBreakerConfig:
failure_threshold: int = 5
recovery_timeout: float = 30.0 # seconds
half_open_max_requests: int = 1
success_threshold: int = 2 # consecutive successes to close
class CircuitBreaker:
def __init__(self, name: str, config: CircuitBreakerConfig = None):
self.name = name
self.config = config or CircuitBreakerConfig()
self.state = CircuitState.CLOSED
self.failure_count = 0
self.success_count = 0
self.last_failure_time = 0.0
self.half_open_requests = 0
self._lock = asyncio.Lock()
async def call(self, func: Callable, fallback: Optional[Callable] = None):
async with self._lock:
if self.state == CircuitState.OPEN:
if time.monotonic() - self.last_failure_time >= self.config.recovery_timeout:
self.state = CircuitState.HALF_OPEN
self.half_open_requests = 0
else:
return await self._handle_fallback(fallback)
if self.state == CircuitState.HALF_OPEN:
if self.half_open_requests >= self.config.half_open_max_requests:
return await self._handle_fallback(fallback)
self.half_open_requests += 1
try:
result = await func()
async with self._lock:
if self.state == CircuitState.HALF_OPEN:
self.success_count += 1
if self.success_count >= self.config.success_threshold:
self._reset()
else:
self.failure_count = 0
self.success_count = 0
return result
except Exception as e:
async with self._lock:
self.failure_count += 1
self.last_failure_time = time.monotonic()
if self.failure_count >= self.config.failure_threshold:
self.state = CircuitState.OPEN
self.success_count = 0
return await self._handle_fallback(fallback)
def _reset(self):
self.state = CircuitState.CLOSED
self.failure_count = 0
self.success_count = 0
self.half_open_requests = 0
async def _handle_fallback(self, fallback):
if fallback:
return await fallback()
raise CircuitBreakerOpenError(f"Circuit breaker '{self.name}' is OPEN")
class CircuitBreakerOpenError(Exception):
passKey differences from the naive version:
- Time-based recovery: After 30 seconds, it automatically tries again.
- Half-open probing: Only lets one request through to test the waters.
- Success threshold: Needs 2 consecutive successes before fully closing.
- Async-first: Because your agents are async, right? Right?
- Fallback support: When the circuit is open, you can return cached data or a default response instead of crashing.
Where This Actually Matters in Multi-Agent Systems
Let’s get concrete. Here’s a real scenario from our platform:
python
# Agent orchestration with circuit breakers
class DocumentProcessingOrchestrator:
def __init__(self):
self.ocr_breaker = CircuitBreaker(
name="ocr_service",
config=CircuitBreakerConfig(
failure_threshold=3,
recovery_timeout=15.0,
half_open_max_requests=1,
success_threshold=2
)
)
self.embedding_breaker = CircuitBreaker(
name="embedding_service",
config=CircuitBreakerConfig(
failure_threshold=5,
recovery_timeout=30.0,
half_open_max_requests=2,
success_threshold=3
)
)
async def process_document(self, doc):
# Step 1: OCR
ocr_result = await self.ocr_breaker.call(
lambda: self.ocr_agent.extract_text(doc),
fallback=lambda: {"text": "", "confidence": 0.0, "source": "fallback"}
)
if ocr_result["confidence"] < 0.5:
# Don't bother with embedding if OCR was garbage
return {"status": "low_confidence", "data": ocr_result}
# Step 2: Embedding
embedding = await self.embedding_breaker.call(
lambda: self.embedding_agent.vectorize(ocr_result["text"]),
fallback=lambda: self._get_stale_embedding(doc.id)
)
return {"status": "success", "ocr": ocr_result, "embedding": embedding}Notice the fallback for the embedding service? That's not just a nice-to-have. When the embedding API is down, we serve stale embeddings from a local cache. The user gets slightly less relevant results instead of a 500 error.
That's the difference between a system that fails gracefully and one that falls over.
The Metrics That Matter
You can't improve what you don't measure. Here's what we track for every circuit breaker:
| Metric | What It Tells You |
|---|---|
| `circuit_breaker_state` | Current state (0=closed, 1=open, 2=half-open) |
| `circuit_breaker_failure_count` | How many consecutive failures |
| `circuit_breaker_trip_count` | Total times circuit has opened |
| `circuit_breaker_fallback_count` | How often fallbacks were used |
| `circuit_breaker_recovery_time` | Time spent in open state |
We push these to Prometheus and alert when any circuit breaker trips more than 5 times in an hour. That's usually a sign of a deeper problem.
Common Mistakes I Still See
1. Global circuit breakers for all agents. Don't do this. Each external dependency should have its own breaker. The OCR service failing shouldn't block the summarization agent that uses a different API.
2. Setting the threshold too high. If you set `failure_threshold` to 50, you've already burned through your error budget before the breaker even trips. Start at 3-5 and tune up.
3. Forgetting about half-open timeouts. If the recovery timeout is 5 minutes, your system is degraded for 5 minutes even if the downstream service recovers in 10 seconds. Keep it short—15-30 seconds is usually right.
4. No fallback strategy. A circuit breaker without a fallback is just a fancy way to throw an exception. Cache, stale data, degraded mode—pick something.
The Real Cost of Getting This Wrong
We onboarded a client from Singapore who had built their own multi-agent system for customer support. No circuit breakers. Just retries with exponential backoff.
When their primary LLM provider had a 4-minute outage, here's what happened:
- 12 agents each retried 8 times before giving up
- Each retry consumed ~2 seconds of timeout
- Total system lockup: 3 minutes and 12 seconds
- Recovery time: 22 minutes (had to restart the orchestrator)
- Lost requests: 847
After we added circuit breakers with 15-second recovery timeouts and local fallback responses, the same outage caused:
- 4 requests served from fallback
- 0 lost requests
- Full recovery in 18 seconds
That's not a 10x improvement. That's a 100x improvement in resilience.
When NOT to Use a Circuit Breaker
Honestly, circuit breakers aren't always the answer. If you're building a simple two-step pipeline where one agent calls another, a timeout with a retry is probably fine.
Circuit breakers shine when:
- You have 5+ agents in a chain
- Agents share dependencies (same API, same database)
- You need to guarantee SLAs
- Your system runs 24/7 and can't have manual intervention
If you're just prototyping, skip the circuit breaker. But the moment you put that system in front of real users, add one.
The Bottom Line
Your multi-agent system is only as strong as its weakest dependency. And dependencies fail. It's not a matter of if, but when.
A circuit breaker doesn't prevent failures. It prevents failures from becoming catastrophes. It gives your system the ability to say "I can't do what you're asking right now, but here's what I can do instead."
That's the difference between a production system and a demo.
Our team in Can Tho has been using this pattern across all our client deployments for the last 18 months. We've seen it save systems that would have otherwise required a full restart. It's not glamorous. But it works.
Stop building houses of cards. Start building systems that bend instead of break.
---
Frequently Asked Questions
What's the difference between a circuit breaker and a retry in multi-agent systems?
A retry assumes the failure is temporary and will succeed if you just try again. A circuit breaker assumes the failure might be prolonged and stops trying entirely to prevent cascading failures. Use retries for idempotent, fast operations. Use circuit breakers for external API calls and long-running agent tasks.
How do I choose the right failure threshold for my circuit breaker?
Start with 3-5 failures within a 30-second window. Monitor your system's normal error rate and tune from there. If you're seeing false positives (circuit tripping during normal operation), increase the threshold. If you're seeing cascading failures before the circuit trips, decrease it.
Can I use circuit breakers with synchronous Python code?
Yes, but you shouldn't. Multi-agent systems are inherently I/O-bound. If you're not using async, you're leaving performance on the table. The synchronous version of the pattern works the same way, but you'll need threading locks instead of async locks.
Should every agent in my system have its own circuit breaker?
No. Each external dependency should have its own circuit breaker. If two agents call the same LLM API, they should share a circuit breaker for that API. If they call different APIs, they need separate breakers. Group by dependency, not by agent.
Related reading: Vietnam Outsourcing: Why It’s the Smartest Offshore Development Move for Tech Leaders in 2025
Related reading: Outsourcing Software Development in 2025: Why Vietnam Is Winning
