Stop Sharing State Like It’s 2019: A Practical Guide to Event Sourcing for Multi-Agent Systems
I’ll be blunt: if your multi-agent system shares state via a global dictionary, a shared Redis key, or—god forbid—a mutable Python object passed between agents, you’re going to have a bad time.
We learned this the hard way. Six months ago, our logistics pipeline had three agents: a router, an optimizer, and a notifier. They shared a single Redis hash for “current order state.” Every 15 minutes, one agent would overwrite another’s data. Orders got lost. Customers got angry. We spent two weeks debugging a race condition that only happened under load.
From Batch to Real-Time: How We Helped a Logistics Startup Track 10,000 Shipments with a Vietnamese AI-Augmented Team
From Batch to Real-Time: How We Helped a Logistics Startup Track 10,000 Shipments with a Vietnamese AI-Augmented Team… ...
The fix? Event sourcing. It’s not new. But for multi-agent systems, it’s the difference between a brittle prototype and a production survivor.
Here’s exactly how we implemented it, the code we used, and why your agents should never, ever share mutable state again.
How We Cut API Response Time by 80% Using Redis, PostgreSQL, and a Vietnamese AI-Augmented Team: A Step-by-Step Performance Optimization Tutorial
How We Cut API Response Time by 80% Using Redis, PostgreSQL, and a Vietnamese AI-Augmented Team Last week,… ...
The Problem: Shared State Is a Liar
Let’s look at what most teams do. They spin up a few agents, each with a reference to the same data store.
python
# Don't do this. Seriously.
class OrderState:
def __init__(self):
self.status = "pending"
self.assigned_agent = None
self.retry_count = 0
state = OrderState() # Shared mutable object
agent_a.update(state)
agent_b.update(state) # Race condition waiting to happenThat’s a time bomb. Agent A reads `status == “pending”`, decides to assign itself, writes `assigned_agent = “agent_a”`. Meanwhile, Agent B reads the same `status`, sees no assigned agent, writes `assigned_agent = “agent_b”`. Now you’ve got two agents both claiming ownership.
This isn’t theoretical. We saw it in production. The logs showed both agents processing the same order simultaneously. The result? Double shipments, angry merchants, and a 3 AM incident call.
Event Sourcing: The Only Sanity Check
Event sourcing flips the model. Instead of storing the current state, you store a sequence of events. Each event is immutable. Each event describes something that *happened*, not something that *is*.
Want to know the current state? Replay the events. It’s that simple.
Here’s the core implementation we use with our Vietnamese team at ECOA AI:
python
from dataclasses import dataclass, field
from typing import List, Dict, Any
from datetime import datetime
import json
@dataclass
class Event:
aggregate_id: str
event_type: str
data: Dict[str, Any]
timestamp: datetime = field(default_factory=datetime.utcnow)
version: int = 0
class EventStore:
def __init__(self):
self.events: List[Event] = []
def append(self, event: Event) -> None:
event.version = len(self.events) + 1
self.events.append(event)
def get_events(self, aggregate_id: str) -> List[Event]:
return [e for e in self.events if e.aggregate_id == aggregate_id]
def replay(self, aggregate_id: str) -> Dict[str, Any]:
state = {"status": "created", "assignments": [], "retries": 0}
for event in self.get_events(aggregate_id):
if event.event_type == "OrderAssigned":
state["assignments"].append(event.data["agent_id"])
state["status"] = "assigned"
elif event.event_type == "OrderProcessed":
state["status"] = "completed"
elif event.event_type == "OrderRetried":
state["retries"] += 1
state["status"] = "retrying"
return stateNotice something? There’s no `update()`. No `set_status()`. You only ever *append*. That’s the whole point.
How Agents Consume Events (Without Fighting)
Agents don’t write to shared state. They read events and emit new ones. Here’s the pattern:
python
class OrderRouterAgent:
def __init__(self, event_store: EventStore):
self.store = event_store
def handle_order(self, order_id: str) -> None:
# Read current state by replaying events
current = self.store.replay(order_id)
if current["status"] != "created":
return # Already handled
# Decide which agent should handle this
agent_id = self.select_agent(current)
# Emit an event, not a state mutation
self.store.append(Event(
aggregate_id=order_id,
event_type="OrderAssigned",
data={"agent_id": agent_id, "reason": "load_balanced"}
))Each agent reads the full event stream, builds its own view of state, and decides what to do next. No two agents ever write to the same row. No locks. No race conditions.
We tested this under 10,000 concurrent orders. Zero state corruption. That’s not a boast—it’s the mathematical guarantee of an append-only log.
Why This Matters for Production Systems
You might be thinking, “But replaying all events to get state is slow.” Fair point. Here’s the fix: snapshots.
python
class SnapshotStore:
def __init__(self):
self.snapshots: Dict[str, Dict[str, Any]] = {}
def save_snapshot(self, aggregate_id: str, state: Dict[str, Any], version: int) -> None:
self.snapshots[aggregate_id] = {"state": state, "version": version}
def get_snapshot(self, aggregate_id: str):
return self.snapshots.get(aggregate_id)
class OptimizedEventStore(EventStore):
def __init__(self, snapshot_frequency: int = 100):
super().__init__()
self.snapshot_store = SnapshotStore()
self.snapshot_frequency = snapshot_frequency
def replay(self, aggregate_id: str) -> Dict[str, Any]:
snapshot = self.snapshot_store.get_snapshot(aggregate_id)
if snapshot:
state = snapshot["state"]
start_version = snapshot["version"]
else:
state = {}
start_version = 0
for event in self.get_events(aggregate_id):
if event.version <= start_version:
continue
self.apply_event(state, event)
return state
def append(self, event: Event) -> None:
super().append(event)
if len(self.get_events(event.aggregate_id)) % self.snapshot_frequency == 0:
state = self.replay(event.aggregate_id)
self.snapshot_store.save_snapshot(event.aggregate_id, state, event.version)We snapshot every 100 events. The replay cost drops to near-zero. In production, our average state reconstruction takes 2-3 milliseconds. That’s faster than a Redis read.
Real Numbers from Our Ho Chi Minh City Team
We paired this architecture with a team of senior developers in Ho Chi Minh City. These engineers understood distributed systems intuitively—they’d seen this pattern in Kafka and EventStore before.
Results after migrating:
- State corruption incidents: 12/month → 0/month
- Debugging time for agent conflicts: 8 hours → 30 minutes
- Throughput: 500 orders/min → 4,500 orders/min (no more lock contention)
- New agent onboarding: 3 days → 4 hours (just read the event stream)
One senior dev on our team put it perfectly: “Event sourcing makes the system explain itself. You don’t guess what happened—you read the log.”
Production Considerations You Can’t Ignore
Event sourcing isn’t magic. You still need to handle:
Idempotency. Agents might crash after writing an event but before acknowledging it. Use a deduplication ID on each event. We use `(agent_id, order_id, attempt_number)` as a unique constraint.
Event versioning. Your event schema will change. We use a `version` field on each event and maintain backward-compatible readers. Old agents read old events, new agents handle both.
Storage costs. Events accumulate. We archive events older than 90 days to cold storage. The snapshot keeps the hot path fast.
Ordering guarantees. If you’re using multiple event store nodes, you need a total order. We use a single-writer pattern with a leader node. Kafka works too, but for 95% of use cases, a single Postgres table with an auto-increment ID is simpler and faster.
The Real Takeaway
Here’s the thing: your multi-agent system will fail. Not if—when. The question is whether it fails gracefully or catastrophically.
Event sourcing gives you a complete audit trail. You can replay any order from any point in time. You can debug production issues by replaying events locally. You can add new agents that learn from historical data.
We’ve been running this in production for four months. Zero state corruption. Zero lost events. Zero “how did that happen?” moments.
That’s the kind of reliability you get when you stop sharing state and start sharing history.
Frequently Asked Questions
Does event sourcing work with real-time agent communication?
Yes, but you need a pub/sub layer on top. We use Redis Streams to push new events to interested agents. The event store is the source of truth; the stream is the notification channel. Agents subscribe to event types they care about and react accordingly.
How do you handle events that fail to process?
We use a dead letter queue pattern. If an agent fails to process an event after 3 retries, it emits an `EventFailed` event with the error details. A supervisor agent monitors these and decides whether to retry, escalate, or skip. The original events are never deleted—they stay in the event store forever.
What’s the storage overhead compared to traditional state storage?
It’s higher, but not as much as you’d think. A typical order generates 5-10 events over its lifetime. Each event is roughly 200 bytes. For 1 million orders, that’s 1-2 GB of event data. Snapshots add another 500 MB. Compare that to the cost of debugging a single production incident, and it’s trivial.
Can I use event sourcing with an existing relational database?
Absolutely. We use Postgres with a single `events` table. The schema is simple: `aggregate_id, event_type, data (JSONB), version, timestamp`. Add an index on `(aggregate_id, version)` and you’re done. We handle 50,000 events/second on a single medium-sized RDS instance. No Kafka required.
Related reading: Why Smart CTOs Hire Vietnamese Developers: A No-Nonsense Strategic Guide
Related reading: Vietnam Outsourcing: The Smart Tech Leader’s Guide to Offshore Development in 2025
Related reading: Outsourcing Software Development: A CTO’s Playbook for Building Remote Teams That Ship
