How to Build a Custom Python Webhook Handler with Idempotency and Retry Logic — A Developer Tutorial

Webhooks are the duct tape of modern APIs. You set one up, events start flowing, and then — without warning — a duplicate arrives. Or your endpoint crashes, and the sender has no idea.

Sound familiar? It’s not a bug. It’s a design problem.

Building a Leaner PR Pipeline with AI Code Review: A Step-by-Step Developer Tutorial

Building a Leaner PR Pipeline with AI Code Review: A Step-by-Step Developer Tutorial I’ve been maintaining open-source projects… ...

At ECOA AI, our Vietnamese engineering teams handle hundreds of webhook integrations per month for clients in the US, Europe, and APAC. We’ve seen every failure mode: duplicate events, lost payloads, race conditions, and silent retries that compound into chaos. Each time, the fix comes down to two patterns: idempotency and retry with exponential backoff.

In this tutorial, you’ll build a Python webhook handler that’s production-ready from the start. No fluff. Just code that scales.

We Slashed a SaaS’s AI Token Costs by 67% with Selective Context Injection — A Vietnam Offshore Case Study

We Slashed a SaaS’s AI Token Costs by 67% with Selective Context Injection — A Vietnam Offshore Case… ...

Python 3.10+
A Redis instance (or you can swap in SQLite for testing)
Basic familiarity with FastAPI (we’ll use it as the web framework)

Why Redis? Because you need fast, atomic checks for idempotency keys. A database works in a pinch, but Redis is lighter and handles concurrency better.

Step 1: The Idempotency Key — Your First Line of Defense

Every webhook payload should carry a unique identifier. If it doesn’t, you generate one from a hash of the event content. We’ll use an `Idempotency-Key` header or a field inside the payload.

python
import hashlib
import json
import uuid

def get_idempotency_key(payload: dict, header_key: str | None = None) -> str:
    if header_key:
        return header_key
    # Fall back to deterministic hash
    raw = json.dumps(payload, sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

You’ll store this key in Redis with a TTL (time-to-live). Why TTL? Because you don’t need to remember old events indefinitely. 24 hours is standard.

python
import redis.asyncio as redis

redis_client = redis.from_url("redis://localhost:6379")

IDEM_TTL = 86400  # 24 hours

async def is_already_processed(key: str) -> bool:
    return await redis_client.get(key) is not None

async def mark_as_processed(key: str, status: str = "completed") -> None:
    await redis_client.setex(key, IDEM_TTL, status)

Now your handler can reject duplicates immediately:

python
@app.post("/webhook")
async def webhook_handler(payload: dict, headers: dict):
    idem_key = get_idempotency_key(payload, headers.get("idempotency-key"))
    if await is_already_processed(idem_key):
        return {"status": "duplicate", "code": 200}  # Idempotent success
    # Process the event
    await process_event(payload)
    await mark_as_processed(idem_key)
    return {"status": "ok"}

This simple check cuts duplicate processing to zero. But what happens when your processing fails?

Step 2: Retry with Exponential Backoff + Jitter

Don’t retry immediately — you’ll just hit the same transient failure. Use a retry decorator with exponential backoff and jitter to spread out retries. We’ll use `tenacity`, a battle-tested Python library.

python
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential(multiplier=2, min=1, max=60),
    retry=retry_if_exception_type((ConnectionError, TimeoutError, HTTPError)),
    reraise=True
)
async def process_event(payload: dict):
    # Simulate processing that might fail
    ...

Here’s the retry schedule:

Attempt	Wait (seconds)
1	1
2	2
3	4
4	8

After the fourth attempt, you stop. But wait — we need to mark the idempotency key as “failed” so you can inspect later.

Step 3: Final Status + Dead Letter Queue

When all retries fail, store the failure and optionally push to a dead letter queue (DLQ) for human review.

python
async def process_event_with_dlq(payload: dict, idem_key: str):
    try:
        await process_event(payload)
        await mark_as_processed(idem_key, "completed")
    except Exception as e:
        await mark_as_processed(idem_key, f"failed: {str(e)}")
        await push_to_dlq(payload, str(e))

A DLQ can be as simple as a Redis list or a database table. Later, you can build a dashboard to replay failed events. Our team in Can Tho built one that cut incident response time by 40% — but that’s another story.

Step 4: Concurrency Safety — The Silent Killer

Here’s where most tutorials stop. But you’re not done. What if two identical webhook requests arrive simultaneously? Both check idempotency, neither finds a key, and both process the event. You’ve lost idempotency.

Fix: use Redis `SET NX` (set if not exists) as a compare-and-swap.

python
async def acquire_lock(key: str, ttl: int = 60) -> bool:
    result = await redis_client.setnx(f"lock:{key}", "1")
    if result:
        await redis_client.expire(f"lock:{key}", ttl)
    return bool(result)

async def release_lock(key: str) -> None:
    await redis_client.delete(f"lock:{key}")

@app.post("/webhook")
async def webhook_handler(payload: dict, headers: dict):
    idem_key = get_idempotency_key(payload, headers.get("idempotency-key"))
    if not await acquire_lock(idem_key):
        return {"status": "processing", "code": 202}  # Tell sender to wait
    try:
        if await is_already_processed(idem_key):
            return {"status": "duplicate", "code": 200}
        await process_event_with_dlq(payload, idem_key)
    finally:
        await release_lock(idem_key)
    return {"status": "ok"}

Now concurrent requests for the same event will see a lock and either wait or return “202 Accepted”. The sender (if well-behaved) will retry with the same idempotency key later.

Putting It All Together

Here’s the full handler — about 60 lines of Python that handle 99% of webhook failure patterns.

python
import hashlib, json, uuid
from fastapi import FastAPI, Request
import redis.asyncio as redis
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

app = FastAPI()
redis_client = redis.from_url("redis://localhost:6379")
IDEM_TTL = 86400

def get_idempotency_key(payload: dict, header_key: str | None = None) -> str:
    if header_key:
        return header_key
    raw = json.dumps(payload, sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

async def acquire_lock(key: str, ttl: int = 60) -> bool:
    res = await redis_client.setnx(f"lock:{key}", "1")
    if res:
        await redis_client.expire(f"lock:{key}", ttl)
    return bool(res)

async def release_lock(key: str) -> None:
    await redis_client.delete(f"lock:{key}")

async def is_processed(key: str) -> bool:
    return await redis_client.get(key) is not None

async def mark_processed(key: str, status: str = "completed") -> None:
    await redis_client.setex(key, IDEM_TTL, status)

@retry(stop=stop_after_attempt(4), wait=wait_exponential(multiplier=2, min=1, max=60), reraise=True)
async def process_event(payload: dict):
    # Your business logic here
    ...

async def push_to_dlq(payload: dict, error: str):
    await redis_client.lpush("dlq", json.dumps({"payload": payload, "error": error}))

async def safe_process(payload: dict, idem_key: str):
    try:
        await process_event(payload)
        await mark_processed(idem_key, "completed")
    except Exception as e:
        await mark_processed(idem_key, f"failed: {str(e)}")
        await push_to_dlq(payload, str(e))

@app.post("/webhook")
async def webhook_handler(request: Request):
    payload = await request.json()
    idem_key = get_idempotency_key(payload, request.headers.get("idempotency-key"))
    if not await acquire_lock(idem_key):
        return {"status": "processing", "code": 202}
    try:
        if await is_processed(idem_key):
            return {"status": "duplicate", "code": 200}
        await safe_process(payload, idem_key)
    finally:
        await release_lock(idem_key)
    return {"status": "ok"}

What About the Sender?

You can’t control your webhook provider. But you can control your side. The patterns above handle duplicates, retries, and concurrency without requiring changes to the upstream API. That’s the kind of defensive coding we teach every developer we onboard in Ho Chi Minh City.

Why This Matters for Your Team

Webhook processing is a common source of data corruption and silent failures. Most teams write a quick endpoint and move on. Then they spend days debugging missing orders or duplicate payments.

We built this pattern into the ECOA AI Platform ACP’s integration layer. Our vetted Vietnamese engineers use a templated version that ships with every new webhook-based integration. The result? 5x fewer webhook-related incidents per client, and response times under 5 minutes when something does break.

If you’re building a team — offshore or local — make sure your developers know these three patterns by heart. It’ll save you a lot of