Build a Custom AI-Powered PR Reviewer with Claude API and GitHub Webhooks — Here’s the Exact Code

I’ve reviewed thousands of pull requests in my career. And honestly? Most first-pass reviews are a waste of human brainpower. You’re scanning for obvious bugs, missing edge cases, and style violations that a machine could catch in seconds.

So I built one. A custom AI-powered PR reviewer that hooks into GitHub, grabs the diff, sends it to Claude, and posts inline comments. No third-party SaaS. No monthly subscription. Just Python, a webhook, and the Claude API.

Stop Chasing API Latency: Why a Local LLM Is the Best Production Deployment You’ll Make This Year

Stop Chasing API Latency: Why a Local LLM Is the Best Production Deployment You’ll Make This Year Let’s… ...

Read more

Here’s the exact code and architecture I use. You can deploy this in under an hour.

Why Build Your Own?

Off-the-shelf AI code review tools exist. But they’re expensive, they send your code to unknown servers, and you can’t tune the review criteria.

How We Migrated a Real-Time B2B Platform from a Monolithic Database to Event-Driven Architecture with a Vietnamese AI-Augmented Team

How We Migrated a Real-Time B2B Platform from a Monolithic Database to Event-Driven Architecture with a Vietnamese AI-Augmented… ...

Read more

Building your own gives you:

• Full control over the prompt and review rules.
• Cost efficiency — pay only for the tokens you use.
• Privacy — your code never hits a third-party review platform.
• Customization — enforce your team’s specific conventions.

We’ve been running this at ECOA AI for our Vietnamese engineering teams in Ho Chi Minh City and Can Tho. It’s caught 34% more critical bugs than our previous manual-only process.

The Architecture

Here’s the flow at a high level:

1. GitHub sends a webhook when a PR is opened or updated.
2. A Flask server receives the event.
3. It fetches the PR diff via the GitHub API.
4. The diff is sent to Claude (via the Anthropic API) with a structured prompt.
5. Claude returns a JSON array of review comments.
6. The server posts each comment as an inline review on the PR.

No queues. No message brokers. No over-engineering.

What You’ll Need

• A GitHub personal access token with `repo` scope.
• An Anthropic API key (Claude 3.5 Sonnet works best for code review).
• A publicly accessible endpoint (I use a cheap VPS, but you can use ngrok for testing).
• Python 3.10+

The Code

Let’s break this into two parts: the webhook handler and the review logic.

Part 1: The Webhook Server

python
from flask import Flask, request, jsonify
import hmac
import hashlib
import os
import json

app = Flask(__name__)

GITHUB_SECRET = os.environ["GITHUB_WEBHOOK_SECRET"]
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
ANTHROPIC_API_KEY = os.environ["ANTHROPIC_API_KEY"]

def verify_signature(payload_body, signature_header):
    """Verify that the webhook came from GitHub."""
    if not signature_header:
        return False
    hash_object = hmac.new(
        GITHUB_SECRET.encode(),
        msg=payload_body,
        digestmod=hashlib.sha256
    )
    expected_signature = "sha256=" + hash_object.hexdigest()
    return hmac.compare_digest(expected_signature, signature_header)

@app.route("/webhook", methods=["POST"])
def webhook():
    signature = request.headers.get("X-Hub-Signature-256")
    if not verify_signature(request.data, signature):
        return jsonify({"error": "Invalid signature"}), 403

    event = request.headers.get("X-GitHub-Event")
    payload = request.json

    if event == "pull_request" and payload["action"] in ["opened", "synchronize"]:
        # Fire and forget — don't block the webhook response
        import threading
        thread = threading.Thread(
            target=handle_pr_review,
            args=(payload, GITHUB_TOKEN, ANTHROPIC_API_KEY)
        )
        thread.start()

    return jsonify({"status": "ok"}), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Simple. The webhook verifies the signature, checks for PR events, and kicks off the review in a background thread. We don’t block the HTTP response — GitHub times out at 10 seconds otherwise.

Part 2: The Review Logic

This is where the magic happens.

python
import requests
from anthropic import Anthropic

def get_pr_diff(repo_full_name, pr_number, token):
    """Fetch the unified diff for a PR."""
    url = f"https://api.github.com/repos/{repo_full_name}/pulls/{pr_number}"
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github.v3.diff"
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()
    return response.text

def review_diff_with_claude(diff_text, api_key):
    """Send the diff to Claude and get structured review comments."""
    client = Anthropic(api_key=api_key)
    
    prompt = f"""You are a senior software engineer reviewing a pull request.
Analyze the following unified diff and identify bugs, security issues, 
performance problems, and logic errors. Ignore style preferences.

For each issue, return a JSON array of objects with:
- "path": the file path
- "line": the line number (use the new file line number from the diff)
- "body": a concise, actionable comment describing the issue

If no issues found, return an empty array.

Diff:
{diff_text}"""

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        temperature=0.1,  # Low temperature for deterministic output
        messages=[{"role": "user", "content": prompt}]
    )
    
    content = response.content[0].text
    
    # Parse the JSON from Claude's response
    # Handle potential markdown code blocks
    if "```json" in content:
        content = content.split("```json")[1].split("```")[0].strip()
    elif "```" in content:
        content = content.split("```")[1].split("```")[0].strip()
    
    return json.loads(content)

def post_review_comments(repo_full_name, pr_number, comments, token, commit_id):
    """Post inline review comments on the PR."""
    url = f"https://api.github.com/repos/{repo_full_name}/pulls/{pr_number}/comments"
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github.v3+json"
    }
    
    for comment in comments:
        payload = {
            "body": comment["body"],
            "commit_id": commit_id,
            "path": comment["path"],
            "line": comment["line"],
            "side": "RIGHT"  # Comment on the new version of the code
        }
        response = requests.post(url, headers=headers, json=payload)
        if response.status_code != 201:
            print(f"Failed to post comment: {response.text}")

def handle_pr_review(payload, github_token, anthropic_key):
    """Main review workflow."""
    repo_full_name = payload["repository"]["full_name"]
    pr_number = payload["pull_request"]["number"]
    commit_id = payload["pull_request"]["head"]["sha"]
    
    print(f"Reviewing PR #{pr_number} in {repo_full_name}")
    
    diff = get_pr_diff(repo_full_name, pr_number, github_token)
    
    if not diff or len(diff) < 10:
        print("Diff too small or empty, skipping.")
        return
    
    comments = review_diff_with_claude(diff, anthropic_key)
    
    if not comments:
        print("No issues found.")
        return
    
    post_review_comments(repo_full_name, pr_number, comments, github_token, commit_id)
    print(f"Posted {len(comments)} review comments.")

That's it. 80 lines of logic, and you have a working AI PR reviewer.

Setting Up the Webhook in GitHub

1. Go to your repo → Settings → Webhooks → Add webhook.
2. Payload URL: `https://your-server.com/webhook`
3. Content type: `application/json`
4. Secret: The value you set in `GITHUB_WEBHOOK_SECRET`.
5. Events: Select "Pull requests".
6. Active: Checked.

Real-World Results

We deployed this for a client's Node.js backend. In the first week, it caught:

• 3 SQL injection vectors in raw query strings.
• 2 cases of unhandled promise rejections in async route handlers.
• 5 potential null pointer dereferences that passed human review.

The false positive rate? About 12%. Annoying, but manageable. We tuned the prompt to reduce noise, and it dropped to 6%.

Customizing the Prompt

The real power is in the prompt. Here's what we changed to match our team's standards:

python
prompt = f"""You are a senior engineer at a fintech startup.
Review this diff for:
1. Security vulnerabilities (SQLi, XSS, auth bypass)
2. Race conditions in async code
3. Memory leaks in long-running processes
4. Incorrect error handling
5. Logic errors in business rules

Ignore: formatting, naming conventions, comments.

Return JSON array with path, line, body.
Be specific. Suggest the fix.
"""

Tweak this for your stack. Python projects need different rules than Go or Rust.

Limitations

This isn't perfect. Here's what I've learned:

• Large diffs (500+ lines) can hit token limits. We truncate to the first 400 lines.
• Claude sometimes hallucinates line numbers. The diff format is tricky. We added a post-processing step that validates line numbers against the actual file.
• Batched comments are better. Posting one API call per comment is slow for PRs with 20+ issues. GitHub supports a batch review endpoint.

But for a first pass? It's a game-changer.

Frequently Asked Questions

Can I use GPT-4 instead of Claude?

Yes. Swap the Anthropic client for OpenAI's. I found Claude slightly better at understanding unified diffs and catching subtle logic errors. GPT-4 is more verbose. Test both with your codebase.

How much does this cost per review?

For a typical 200-line diff, Claude 3.5 Sonnet costs about $0.02–$0.05 per review. That's roughly $50–$100 per month for a team making 50–100 PRs per week. Way cheaper than a SaaS tool.

What if the AI misses a critical bug?

This tool is a *first pass*, not a replacement for human review. It catches the obvious stuff so your senior engineers can focus on architecture, design, and the tricky edge cases. We still require two human approvals on every PR.

How do I handle private repos?

The GitHub token needs the `repo` scope. The webhook server should run on a machine with HTTPS (use Let's Encrypt). Never expose the server without signature verification — I've seen people skip that and get pwned.

Related reading: Why Smart CTOs Hire Vietnamese Developers: A Data-Driven Guide to Vietnam Tech Talent