I Benchmarked 5 Python Async Patterns on a 10K-Request Pipeline — Here’s What Actually Survived Production

You’ve read the tutorials. You’ve seen the pretty diagrams. But when your pipeline needs to handle 10,000 concurrent API requests without falling over, which async pattern actually works?

I spent two weeks testing this on a real data ingestion pipeline for a logistics client. We had a Vietnamese team in Ho Chi Minh City building the core, and I was responsible for the concurrency architecture. We tried five different patterns. Only one made it to production without a catastrophic failure.

State Management Is the Silent Killer of Multi-Agent Systems: Here’s How We Fixed It

State Management Is the Silent Killer of Multi-Agent Systems: Here’s How We Fixed It You’ve built a shiny… ...

Read more

Here’s what we learned.

The Test Setup

We built a synthetic pipeline that simulates what our logistics client actually does: fetch order data from an external API, transform it, and write it to a PostgreSQL database. Each request takes about 200ms of simulated I/O wait (network + database write).

We Migrated a 10TB Kafka Cluster Without a Single Message Lost: What We Learned With a Vietnam-Based Team

We Migrated a 10TB Kafka Cluster Without a Single Message Lost: What We Learned With a Vietnam-Based Team… ...

Read more

The target: 10,000 requests in under 60 seconds.

The hardware: A single 8-core machine with 16GB RAM. No distributed magic. Just raw Python concurrency.

We measured:

• Total execution time (seconds)
• Memory usage (peak RSS)
• Error rate (failed requests)
• CPU utilization

Pattern 1: Vanilla asyncio with `asyncio.gather`

This is the default recommendation. It’s simple. It’s elegant. It’s also dangerously misleading.

python
import asyncio
import aiohttp

async def process_order(order_id: int):
    async with aiohttp.ClientSession() as session:
        async with session.get(f"https://api.example.com/orders/{order_id}") as resp:
            data = await resp.json()
    # Simulate DB write
    await asyncio.sleep(0.1)
    return data

async def main():
    tasks = [process_order(i) for i in range(10000)]
    results = await asyncio.gather(*tasks, return_exceptions=True)

Results:

• Total time: 34.2 seconds
• Peak memory: 1.2 GB
• Error rate: 0.2% (20 requests failed due to connection timeouts)

Looks good, right? Wrong. The problem is that `asyncio.gather` creates all 10,000 tasks at once. The event loop becomes a bottleneck. You’re hammering the external API with 10,000 concurrent connections. Most APIs will throttle you. Ours did.

Verdict: Works for <500 concurrent tasks. Beyond that, you're asking for trouble.

Pattern 2: asyncio with Semaphore (Rate Limiting)

Let’s fix the obvious problem. Add a semaphore to limit concurrency.

python
semaphore = asyncio.Semaphore(100)

async def process_order_with_limit(order_id: int):
    async with semaphore:
        async with aiohttp.ClientSession() as session:
            async with session.get(f"https://api.example.com/orders/{order_id}") as resp:
                data = await resp.json()
        await asyncio.sleep(0.1)
        return data

Results:

• Total time: 52.8 seconds
• Peak memory: 480 MB
• Error rate: 0%

Better memory. Zero errors. But 50% slower. Why? Because we’re limited to 100 concurrent requests, and each takes ~300ms. Math: 10,000 / 100 * 0.3 = 30 seconds just for the I/O. Plus overhead.

Verdict: Safe, predictable, but slow for high-throughput scenarios.

Pattern 3: Multiprocessing with `concurrent.futures.ProcessPoolExecutor`

Let’s throw more CPU cores at it. Because more cores = faster, right?

python
from concurrent.futures import ProcessPoolExecutor
import requests

def process_order_sync(order_id: int):
    resp = requests.get(f"https://api.example.com/orders/{order_id}")
    data = resp.json()
    # Simulate DB write
    time.sleep(0.1)
    return data

with ProcessPoolExecutor(max_workers=8) as executor:
    results = list(executor.map(process_order_sync, range(10000)))

Results:

• Total time: 48.5 seconds
• Peak memory: 2.8 GB
• Error rate: 3.5%

Oh, the memory. Each process has its own Python interpreter. That’s 8 copies of your entire application. And the error rate? Socket exhaustion. Each process opens its own connections without coordination.

Verdict: Great for CPU-bound tasks. Terrible for I/O-bound network pipelines.

Pattern 4: ThreadPoolExecutor with Queue

Old school. Let’s use threads and a producer-consumer pattern.

python
from concurrent.futures import ThreadPoolExecutor
from queue import Queue
import threading

def worker(queue: Queue, results: list):
    while True:
        order_id = queue.get()
        if order_id is None:
            break
        resp = requests.get(f"https://api.example.com/orders/{order_id}")
        data = resp.json()
        results.append(data)
        queue.task_done()

queue = Queue()
for i in range(10000):
    queue.put(i)

results = []
threads = [threading.Thread(target=worker, args=(queue, results)) for _ in range(100)]
for t in threads:
    t.start()
queue.join()

Results:

• Total time: 44.1 seconds
• Peak memory: 680 MB
• Error rate: 1.2%

Better than multiprocessing on memory, but the GIL becomes a problem. Python’s Global Interpreter Lock means only one thread executes Python bytecode at a time. Your I/O is non-blocking, but the GIL still adds overhead.

Verdict: Works, but you’re fighting the GIL. Not ideal for high concurrency.

Pattern 5: Hybrid Multi-Agent Orchestration with Redis Streams

This is where things get interesting. We used a pattern that combines lightweight async workers with a Redis Stream as the coordination layer. Each worker is an independent async process that reads from the stream, processes, and writes results.

python
import asyncio
import aioredis
import aiohttp

class OrderWorker:
    def __init__(self, worker_id: int, stream_key: str):
        self.worker_id = worker_id
        self.stream_key = stream_key
        self.redis = None
        
    async def start(self):
        self.redis = await aioredis.from_url("redis://localhost")
        while True:
            # Blocking read from stream
            result = await self.redis.xreadgroup(
                "order_workers", 
                f"worker_{self.worker_id}",
                {self.stream_key: ">"},
                count=10,
                block=1000
            )
            if result:
                for stream_name, messages in result:
                    for msg_id, msg_data in messages:
                        order_id = int(msg_data[b'order_id'])
                        await self.process_order(order_id)
                        await self.redis.xack(self.stream_key, "order_workers", msg_id)

    async def process_order(self, order_id: int):
        async with aiohttp.ClientSession() as session:
            async with session.get(f"https://api.example.com/orders/{order_id}") as resp:
                data = await resp.json()
        await asyncio.sleep(0.1)
        # Write result back to another stream
        await self.redis.xadd("order_results", {"order_id": order_id, "data": str(data)})

We deployed 20 workers across 4 processes (5 workers each). Each worker is a lightweight async task that pulls from Redis Stream.

Results:

• Total time: 28.4 seconds
• Peak memory: 520 MB
• Error rate: 0.02% (2 failures, both due to network blips)
• Survived production? Yes. Running for 6 months without a single crash.

Why did this win? Three reasons:

1. Backpressure handling: Redis Streams naturally handle backpressure. If workers are slow, messages queue up. No data loss.
2. Graceful degradation: If a worker crashes, Redis re-delivers the message to another worker. No manual retry logic needed.
3. Horizontal scaling: We can add more workers without changing code. Just spin up another process.

The Hard Truth

Most async tutorials are lying to you. They show `asyncio.gather` with 10 tasks and call it a day. Real production pipelines need:

• Rate limiting (semaphores or connection pools)
• Backpressure handling (queues or streams)
• Graceful failure recovery (retry logic with exponential backoff)
• Observability (metrics on queue depth, processing time, error rates)

The hybrid Redis Stream pattern gives you all of this with minimal code. It’s not the fastest theoretical pattern (that’s vanilla asyncio), but it’s the most resilient pattern.

What We Actually Deployed

For our logistics client, we ended up with a system that runs 15 worker processes on 3 machines. Each process runs 10 async workers. That’s 150 concurrent workers consuming from a single Redis Stream.

The client processes 50,000 orders per hour during peak. Latency is under 500ms for 99th percentile. Zero data loss in 6 months.

And honestly? That’s the metric that matters. Not theoretical throughput. Not academic benchmarks. Production survival.

Key Takeaways

The team in Vietnam actually suggested the Redis Stream pattern. They’d used it before for a fintech project in Can Tho. I was skeptical at first. “Redis for orchestration? That’s overkill.” I was wrong. Dead wrong.

Sometimes the best pattern is the one your offshore team has already battle-tested.

—

Frequently Asked Questions

Is vanilla asyncio ever suitable for production pipelines?

Yes, but only for low-concurrency scenarios (under 500 concurrent tasks) where you control both the client and server. If you’re hitting external APIs or databases, you’ll hit connection limits. Always add a semaphore or connection pool limiter.

Why not use Celery instead of Redis Streams?

Celery adds significant complexity (message broker, result backend, worker management). Redis Streams give you the same core functionality (reliable message delivery, consumer groups, acknowledgment) with less overhead. For pipelines under 100K requests/day, Redis Streams are simpler and faster.

How do you handle worker crashes with Redis Streams?

Redis Streams use consumer groups with pending message lists. If a worker crashes without acknowledging a message, Redis marks it as “pending” and re-delivers it to another worker after a timeout. Set `autoclaim` to handle re-delivery automatically.

What about CPU-bound tasks in the pipeline?

For CPU-bound work, use a separate process pool within each async worker. We use `concurrent.futures.ProcessPoolExecutor` with 2-4 workers per async process. This keeps the event loop responsive while still leveraging multiple cores.

Related reading: Why You Should Hire Vietnamese Developers: The Smart Offshore Play for 2025

Related reading: Vietnam Outsourcing: The Smartest Move for Your Offshore Development in 2025