Build a Custom AI-Powered PR Reviewer with Claude API and GitHub Webhooks: Here’s the Exact Code

Let’s be real. Code reviews are a bottleneck. They’re the part of the workflow where your senior engineers get pulled into 30-minute loops over a missing semicolon or a poorly named variable. I’ve seen it happen too many times.

But here’s the thing: you don’t need to replace your human reviewers. You need to give them a better first pass.

Your Open Source PRs Are Getting Rejected: The Real Data on Why (And a Practical Fix for Each)

Your Open Source PRs Are Getting Rejected: The Real Data on Why (And a Practical Fix for Each)… ...

Read more

At ECOA AI, we built a custom AI-powered PR reviewer that runs on every pull request. It catches the low-hanging fruit—style violations, missing error handling, security smells—before a human even looks at the diff. The result? Our senior devs spend 60% less time on reviews. They focus on architecture, not formatting.

And no, we didn’t use some off-the-shelf SaaS product. We built it ourselves with the Claude API and GitHub webhooks.

Why You Should Hire Vietnamese Developers: A Strategic Guide for Tech Leaders

TL;DR: Vietnam is emerging as a premier offshore destination for software development, offering a unique blend of technical… ...

Read more

Here’s the exact code, the architecture, and the lessons we learned running this in production for our Vietnam-based engineering teams in Ho Chi Minh City and Can Tho.

Why Build a Custom AI-Powered PR Reviewer?

You might be thinking, “Why not just use GitHub Copilot for PRs or CodeRabbit?” Good question.

The short answer: control and context.

Off-the-shelf tools don’t know your codebase’s specific conventions. They don’t know your team’s style guide. They don’t know that your team in Can Tho uses a specific naming convention for database migrations. A custom solution lets you inject that context.

Plus, you own the data. Your code never leaves your infrastructure. For clients in fintech or healthcare, that’s non-negotiable.

The Architecture: Webhooks, Actions, and Claude

Here’s the flow we use:

1. A developer opens a PR on GitHub.
2. A webhook fires a payload to our backend.
3. Our service fetches the diff from GitHub’s API.
4. We chunk the diff into manageable pieces.
5. Each chunk goes to Claude (via API) with a strict prompt.
6. Claude returns structured feedback.
7. We post the review as a PR comment.

It’s not complicated. But the details matter.

Step 1: Set Up the GitHub Webhook

First, you need a webhook endpoint. We run this as a simple FastAPI service. Here’s the skeleton:

python
from fastapi import FastAPI, Request, HTTPException
import hmac
import hashlib
import os

app = FastAPI()

GITHUB_SECRET = os.getenv("GITHUB_WEBHOOK_SECRET")

@app.post("/webhook")
async def webhook(request: Request):
    body = await request.body()
    signature = request.headers.get("x-hub-signature-256")
    
    if not verify_signature(body, signature):
        raise HTTPException(status_code=403, detail="Invalid signature")
    
    payload = await request.json()
    
    if payload.get("action") in ["opened", "synchronize"]:
        pr_data = extract_pr_data(payload)
        # Kick off async review
        asyncio.create_task(run_review(pr_data))
    
    return {"status": "ok"}

Don’t skip the signature verification. Seriously. We learned that one the hard way after a bot sent us 1,200 fake webhooks in 10 minutes.

Step 2: Fetch the Diff

Once you have the PR number and repo details, fetch the diff:

python
import httpx

async def fetch_diff(repo_full_name: str, pr_number: int, token: str) -> str:
    url = f"https://api.github.com/repos/{repo_full_name}/pulls/{pr_number}"
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github.v3.diff"
    }
    
    async with httpx.AsyncClient() as client:
        response = await client.get(url, headers=headers)
        response.raise_for_status()
        return response.text

The diff comes back as plain text. For a typical feature branch, you’re looking at 200-800 lines of diff. That’s manageable. For a massive refactor? You’ll need to chunk it.

Step 3: Chunk the Diff Intelligently

Claude has a context window, but throwing an entire 2,000-line diff at it in one shot is wasteful and expensive. You’ll also get worse results.

We chunk by file:

python
import re

def chunk_diff(diff_text: str) -> list[dict]:
    chunks = []
    current_file = None
    current_lines = []
    
    for line in diff_text.split("\n"):
        if line.startswith("diff --git"):
            if current_file and current_lines:
                chunks.append({"file": current_file, "diff": "\n".join(current_lines)})
            current_file = re.search(r"a/(.+) b/.+", line).group(1)
            current_lines = [line]
        else:
            current_lines.append(line)
    
    if current_file and current_lines:
        chunks.append({"file": current_file, "diff": "\n".join(current_lines)})
    
    return chunks

This gives you one chunk per file. Each chunk gets its own Claude call. You can parallelize these calls for speed.

Step 4: Build the Claude Prompt

This is where the magic happens. Your prompt needs to be strict. Claude will drift if you give it too much freedom.

Here’s the prompt we use:

python
SYSTEM_PROMPT = """You are an expert code reviewer. Your job is to analyze pull request diffs and provide actionable feedback.

Rules:
1. Only flag issues that are objectively wrong or will cause bugs.
2. Do not comment on style unless it violates a common security practice.
3. For each issue, provide the exact line number, a severity level (CRITICAL, WARNING, INFO), and a suggested fix.
4. If the code is clean, say nothing.
5. Output in JSON format only.

Output format:
{
  "reviews": [
    {
      "line": 42,
      "severity": "WARNING",
      "message": "This SQL query is vulnerable to injection. Use parameterized queries instead.",
      "suggestion": "cursor.execute('SELECT * FROM users WHERE id = %s', (user_id,))"
    }
  ]
}

If no issues found, return {"reviews": []}"""

USER_PROMPT = f"""Review this diff for file {file_path}:

{diff_content}"""

We use `claude-sonnet-4-20250514` for this. It’s fast enough (2-3 seconds per file) and accurate. The cost? About $0.03 per review for a typical PR with 5-8 files changed.

Step 5: Post the Review to GitHub

Once Claude returns the JSON, we post it as a PR review comment:

python
async def post_review(repo_full_name: str, pr_number: int, reviews: list, token: str):
    if not reviews:
        return
    
    body = "## AI Review Results\n\n"
    for review in reviews:
        emoji = {"CRITICAL": "🔴", "WARNING": "🟡", "INFO": "🔵"}
        body += f"{emoji.get(review['severity'], '⚪')} **Line {review['line']}** ({review['severity']}): {review['message']}\n\n"
        if review.get('suggestion'):
            body += f"```suggestion\n{review['suggestion']}\n```\n\n"
    
    url = f"https://api.github.com/repos/{repo_full_name}/pulls/{pr_number}/reviews"
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github.v3+json"
    }
    
    payload = {
        "body": body,
        "event": "COMMENT"
    }
    
    async with httpx.AsyncClient() as client:
        await client.post(url, json=payload, headers=headers)

We use `event: “COMMENT”` instead of `APPROVE` or `REQUEST_CHANGES`. The AI is advisory, not authoritative. Human reviewers still make the final call.

Production Lessons from Running This for 6 Months

We’ve been running this system for our Vietnamese development teams since January. Here’s what we learned:

Lesson 1: Claude hallucinates line numbers. About 8% of the time, the line numbers in the diff don’t match the actual file. We added a validation step that checks if the line exists in the file. If it doesn’t, we drop the comment.

Lesson 2: Chunk ordering matters. Sending files in dependency order (models before controllers, for example) gives Claude better context. We sort chunks by file type: configs first, then models, then services, then controllers.

Lesson 3: Rate limits are real. GitHub’s API allows 5,000 requests per hour. Claude’s API has its own limits. We added a simple queue with Redis to handle bursts:

python
import redis.asyncio as redis

class ReviewQueue:
    def __init__(self):
        self.redis = redis.from_url("redis://localhost:6379")
    
    async def enqueue(self, pr_data: dict):
        await self.redis.rpush("review_queue", json.dumps(pr_data))
    
    async def dequeue(self):
        data = await self.redis.lpop("review_queue")
        return json.loads(data) if data else None

Lesson 4: Not all PRs need AI review. We skip PRs that are:

• Less than 10 lines changed (trivial)
• Only markdown or documentation changes
• From bots (Dependabot, Renovate)

This cut our API costs by 40%.

The Results: What We Measured

After 6 months and 1,847 PRs reviewed, here’s the data:

That 14% false positive rate is annoying, but we’re improving it. We added a feedback loop: if a human dismisses an AI comment, we log that and use it to fine-tune the prompt.

Deploying This with a Vietnam-Based Team

Here’s the kicker. We built this system with a team of 4 developers in Can Tho, Vietnam. Two juniors, two mids. Cost? $6,000/month total.

They didn’t just build the AI reviewer. They also:

• Set up the CI/CD pipeline on GitHub Actions
• Wrote integration tests for the webhook handler
• Built a dashboard in Grafana showing review metrics
• Handled all the edge cases (empty diffs, binary files, merge conflicts)

That’s the ECOA AI advantage. You get the code, the architecture, and a team that ships.

Ready to Build Your Own?

The code above is production-ready. You can adapt it for your stack in an afternoon. But if you want to skip the setup and get a team that’s already done this 1,800+ times, we’re here.

Our developers in Ho Chi Minh City and Can Tho know this system inside out. They can deploy it for your repo in 48 hours.

Now go build something that makes your senior engineers actually enjoy code reviews again.

—

Frequently Asked Questions

Q: How much does the Claude API cost for this setup?

For a typical PR with 5-8 files changed, expect $0.02 to $0.05 per review. At 100 PRs per week, that’s about $15/month. The GitHub API calls are free within the standard rate limits.

Q: Can I use a different AI model instead of Claude?

Absolutely. The architecture is model-agnostic. Swap the Claude API call for OpenAI’s GPT-4o or even a local model via Ollama. We chose Claude because it follows structured output instructions more reliably for code review tasks.

Q: How do you handle binary files or large diffs?

We skip binary files entirely. For large diffs (over 500 lines), we split them into logical sections and process each section independently. If a single file has more than 300 lines changed, we only review the first 300 lines. The remaining lines get flagged for human review.

Q: Does this replace human code review entirely?

No. And it shouldn’t. The AI catches surface-level issues—missing error handling, security vulnerabilities, convention violations. Human reviewers focus on architecture, design patterns, and business logic. Think of the AI as a first-pass filter, not a replacement.

Related reading: Outsourcing software development in 2025: A CTO’s playbook for smart execution