Your Multi-Agent System Has a Scheduling Problem: How Priority Queuing Slashed Our Critical Task Latency by 58%

I’ll admit it. We were proud of our multi-agent system. Seven specialized agents, event-driven orchestration, graceful error recovery — the works.

But something felt off.

How AI Is Reshaping the Software Development Lifecycle (And Why It Matters)

TL;DR: AI is transforming every phase of the software development lifecycle—from planning to deployment—enabling teams to ship faster,… ...

Read more

High-priority tasks — the ones that directly impacted paying customers — were sitting in queues behind batch processing jobs and internal analytics. A financial reconciliation request from a user would queue up right next to “scrape competitor blog for new posts.”

That’s not just inefficient. It’s a business liability.

Vietnam Outsourcing: Why Your Next Offshore Development Team Should Be in Hanoi or Ho Chi Minh City

TL;DR: Vietnam is quietly becoming the top destination for software outsourcing in Southeast Asia. Lower costs than China,… ...

Read more

Here’s the hard truth: most multi-agent orchestrators use FIFO (first-in, first-out) queuing by default. Redis Streams, RabbitMQ, Kafka — they’ll all happily process tasks in arrival order unless you explicitly tell them otherwise. And when you’re running 10+ agent types with wildly different SLAs, FIFO is a quiet killer.

We fixed this with a priority-based scheduling layer. Here’s the exact architecture, the code patterns that matter, and the results we measured.

The Default Scheduler Is a Liar

Most teams building multi-agent systems start with a simple work queue. An orchestrator pushes tasks, agents pull them, everyone goes home happy.

But in production, tasks aren’t equal. Consider this:

• `Task A`: Process a customer payment dispute (SLA: 30 seconds)
• `Task B`: Generate a weekly PDF report (SLA: 2 hours)
• `Task C`: Re-index the search cache (SLA: 15 minutes)

Under a naive FIFO queue, if 50 `Task B` items arrive just before a `Task A`, your customer waits minutes for a response that should take seconds.

We ran the numbers on our production logs. 42% of our critical-path tasks were delayed by at least 3x their expected duration because they queued behind lower-priority work. That’s not a scheduling bug — that’s a design flaw.

The Architecture: Priority Queues + Weighted Agent Pull

We didn’t rip out Redis Streams. We built a thin scheduling layer on top of it. Here’s the structure:

┌─────────────────┐
│   Orchestrator  │
│  (Task Producer) │
└────────┬────────┘
         │
         ▼
┌──────────────────────────────────┐
│    Priority Router (New Layer)    │
│  ┌──────┐ ┌──────┐ ┌──────┐    │
│  │ P0   │ │ P1   │ │ P2   │    │
│  │ Queue│ │ Queue│ │ Queue│    │
│  └──┬───┘ └──┬───┘ └──┬───┘    │
│     │        │        │        │
│     └────────┴────────┘        │
│              │                 │
│       Weighted Dispatcher      │
└──────────────────────────────────┘
         │
         ▼
┌─────────────────┐
│   Agent Pool    │
│ (7 agent types) │
└─────────────────┘

Three priority tiers: P0 (customer-facing, <30s SLA), P1 (operational, <5min SLA), P2 (batch, best-effort). The router maintains three Redis Streams — one per tier.

The key innovation is the weighted dispatcher. Agents don’t pull from a single stream. They pull from all three, but with a probability weighting: P0 gets 70% of agent attention, P1 gets 25%, P2 gets 5%.

This isn’t strict priority (which can starve lower tiers entirely). It’s *weighted starvation prevention*.

The Code: Weighted Agent Dispatcher

Here’s the core logic we run inside each agent worker. It’s a Python class that replaces the naive `XREAD` loop:

python
import random
import time
from redis import Redis

class PriorityDispatcher:
    def __init__(self, redis_client: Redis, weights=None):
        self.redis = redis_client
        # P0 gets 70% of poll attempts, P1 gets 25%, P2 gets 5%
        self.weights = weights or {"p0": 0.70, "p1": 0.25, "p2": 0.05}
        self.streams = {
            "p0": "tasks:p0",
            "p1": "tasks:p1",
            "p2": "tasks:p2",
        }
        # Track last read IDs per stream
        self.last_ids = {k: "$" for k in self.streams}
    
    def next_task(self, timeout_ms=2000):
        """Get the next task using weighted priority selection."""
        # Weighted random choice of which stream to poll
        tiers = list(self.weights.keys())
        probs = list(self.weights.values())
        selected = random.choices(tiers, weights=probs, k=1)[0]
        
        # Try the selected stream first
        result = self._try_read(self.streams[selected], timeout_ms)
        if result:
            return result
        
        # Fallback: check P0 aggressively if we didn't hit it
        if selected != "p0":
            result = self._try_read(self.streams["p0"], timeout_ms // 2)
            if result:
                return result
        
        # Final fallback: any tier
        for tier in ["p0", "p1", "p2"]:
            result = self._try_read(self.streams[tier], 100)
            if result:
                return result
        
        return None
    
    def _try_read(self, stream_key, timeout_ms):
        result = self.redis.xread(
            {stream_key: self.last_ids[stream_key]},
            count=1,
            block=timeout_ms,
        )
        if result:
            stream_name, messages = result[0]
            if messages:
                msg_id, msg_data = messages[0]
                self.last_ids[stream_key] = msg_id
                return {stream_key: {msg_id: msg_data}}
        return None

The key insight? That `random.choices()` call with weighted probabilities. It ensures P0 tasks get the most attention without creating a hard lockout on lower tiers.

But wait — there’s a subtle issue. If P0 is consistently busy, P2 tasks can still get starved over long periods. We solved that with a backpressure timeout: if any P2 task waits longer than 10 minutes, its priority automatically escalates to P1.

Priority Escalation via TTL

Each task gets a `priority_ttl` field at creation. A background sweeper checks every 30 seconds:

python
def escalate_stale_tasks(redis: Redis):
    for stream_key in ["tasks:p2", "tasks:p1"]:
        tasks = redis.xrange(stream_key, min="-", max="+", count=50)
        for msg_id, msg_data in tasks:
            created_at = float(msg_data.get(b"created_at", 0))
            priority_ttl = int(msg_data.get(b"priority_ttl_s", 600))
            if time.time() - created_at > priority_ttl:
                target = "tasks:p1" if stream_key == "tasks:p2" else "tasks:p0"
                # Move task to higher priority stream
                redis.xadd(target, msg_data)
                redis.xdel(stream_key, msg_id)

This guarantees that no task gets permanently stuck in a low-priority queue. It’s a safety net, not a crutch, but essential for production.

The Results: Real Numbers from Production

We deployed this scheduler across our multi-agent system serving a fintech reconciliation pipeline. The system handles about 4,000 tasks per hour across 7 agent types.

Critical tasks got almost 3x faster. Batch tasks took a small hit — but still well within their SLA.

The 1.1% P2 timeout rate is worth calling out. That’s the cost of prioritization. We addressed it by tuning the escalation TTL from 10 minutes down to 7, which brought P2 timeouts back below 0.5%.

Why Most Teams Miss This

Honestly, I think it’s because priority scheduling sounds “simple” in theory. “Just use multiple queues!” everyone says. But the devil’s in the weighted dispatch and escalation mechanics.

Most Redis Streams tutorials show you `XREADGROUP` with a single stream. They don’t talk about *which* stream your agent should read from at any given moment. And when your agent pool is distributed across time zones — our team has developers in Ho Chi Minh City and Can Tho — a naive round-robin across streams wastes cycles on empty queues.

The weighted random approach adapts to traffic naturally. When P0 is loaded, agents hit it more often. When P0 is quiet, agents automatically shift to lower tiers. No central scheduler needed.

A Practical Pattern, Not a Framework

We didn’t build a framework. We built a pattern — about 120 lines of Python that sits between Redis and our agent workers. That’s it. You don’t need a new orchestration platform. You need a smarter way to pull tasks.

Here’s the checklist if you want to implement this:

1. Categorize your tasks into 3 priority tiers at the producer level
2. Route to separate Redis Streams per tier (or Kafka topics, or RabbitMQ queues)
3. Implement weighted agent polling with `random.choices()` and probability weights
4. Add escalation logic — stale tasks auto-upgrade to higher tiers
5. Monitor per-tier latency — not just average queue depth

Skip step 4 and you’ll see P2 starvation within a week. Skip step 3 and you might as well use FIFO.

When Priority Becomes Political

Here’s something nobody talks about: once you introduce priority queues, every team wants P0 for their tasks.

We had a product manager insist that weekly analytics reports should be P0 “because the CEO reads them.” Sorry, but no. Priority should map to *customer-facing latency sensitivity*, not internal optics. We set up a simple rubric:

• P0: User waits for this. Directly blocks a customer action.
• P1: Operational. User doesn’t wait, but system depends on it.
• P2: Batch. No user impact for delays under 30 minutes.

That rubric killed the political battles. Mostly.

Frequently Asked Questions

Q: Does this pattern work with Kafka instead of Redis Streams?

Yes, but you’ll need consumer groups per partition (one per priority tier) and a weighted `poll()` strategy in your consumer. Kafka’s `pause()` and `resume()` on specific partitions can also work — but it’s more complex than Redis Streams. We chose Redis because our task volume (4K/hr) didn’t warrant Kafka’s overhead.

Q: Won’t weighted polling still starve P2 tasks under sustained P0 load?

Related reading: Why Vietnam Outsourcing Is the Smartest Move for Your Tech Stack in 2025