AI Code Review Benchmark 2026: Every Tool Tested, One Honest Comparison

AI Code Review Benchmark 2026: Every Tool Tested, One Honest Comparison

Five different organizations published AI code review benchmarks in the past six months. Each one crowned a different winner. Greptile measured an 82% catch rate — for themselves. When Augment tested Greptile on the same five repositories, the number dropped to 45%. CodeRabbit scored 44% in one benchmark and 51.2% in another. Qodo claimed 60.1% F1, the highest published score, on their own test suite.

None of these numbers are wrong. They just measured different things, scored differently, and — in every case — the organization running the test happened to win.

We collected every published benchmark, put the data side by side, and found the patterns hiding underneath the contradictions. This page is the result: one comparison table, a breakdown of why results diverge, and the methodology for an open benchmark anyone can reproduce.

Every AI code review benchmark in one table

Nobody has combined these results before. Here they are, sorted by the organization that ran each test:

Martian Benchmark (February 2026) — Independent

The closest thing to a neutral evaluation. Martian was founded by researchers from DeepMind, Anthropic, and Meta. They open-sourced their dataset, judge prompts, and evaluation pipeline.

Scale: Offline test on 50 PRs + online monitoring of real GitHub activity (Jan–Feb 2026). The online component tracked whether developers actually fixed code after AI comments — a behavioral signal that is harder to game than synthetic testing.

Scoring: A comment counted as useful if the developer changed code in response. Comments that developers ignored or dismissed scored against the tool.

Greptile Benchmark (2025) — Vendor

Greptile tested five tools on 50 PRs across five popular open-source repositories: Sentry (Python), Cal.com (TypeScript), Grafana (Go), Keycloak (Java), and Discourse (Ruby).

What "catch rate" means here: A bug counted as "caught" only when the tool explicitly identified the faulty code in a line-level comment and explained the impact. Default settings were used for all tools — no custom rules.

The problem: When Augment later tested the same five repositories, Greptile scored 45% — not 82%. Same repos, dramatically different results. The gap reveals how much the definition of "caught" and the specific bugs chosen can swing results.

Augment Code Benchmark (2026) — Vendor

Augment tested seven tools on the same five repositories as Greptile but expanded the ground truth dataset with manual verification.

Key insight: Precision and recall tell different stories. Claude Code had the second-highest recall (51% — it found a lot of bugs) but the lowest precision (23% — most of its comments were false positives). CodeRabbit showed the inverse pattern in Augment's test: reasonable recall, low precision.

Qodo Benchmark (2026) — Vendor

Qodo tested their multi-agent approach against eight tools across 100 PRs by injecting complex defects into real-world merged pull requests from active open-source repositories.

Methodology: Qodo's injection approach inserts realistic bugs that simultaneously test code correctness and code quality. They designed defects to require understanding of the broader codebase, not just the diff — pushing tools toward deeper analysis.

CodeAnt AI Benchmark (2026) — Vendor

CodeAnt published results from a benchmark claiming 200,000 real pull requests — by far the largest dataset.

Scale claim: 200K PRs is dramatically larger than other benchmarks (50-100 PRs). However, the methodology details and raw data have not been published for independent verification.

Why every vendor wins their own benchmark

The pattern is consistent: Greptile tested Greptile and won. Augment tested Augment and won. Qodo tested Qodo and won. Three mechanisms explain this without assuming intentional manipulation:

1. The "caught" definition varies

Greptile counted a bug as caught when the tool identified the faulty code in a line-level comment. Augment counted precision and recall separately, penalizing noise. Martian used developer behavior — did someone actually fix the code? Three definitions, three different scores for the same tool on the same repository.

A tool that posts twenty comments per PR will score high on recall (it found the bug somewhere in those twenty comments) but low on precision (nineteen of those comments were noise). Depending on which metric you weight, the same tool looks excellent or mediocre.

2. Bug selection bias

Each vendor selects or injects bugs that align with their tool's strengths. A tool optimized for security vulnerabilities will look brilliant on a bug set heavy with injection flaws — and mediocre on a set dominated by logic errors or race conditions. Nobody publishes a benchmark where they lose.

3. Small sample sizes amplify noise

Fifty PRs spread across five repos means ten PRs per repo. At that scale, one ambiguous edge case can swing a tool's score by several percentage points. Greptile's 82% vs Augment's 45% on the same tool is statistically possible with 50 samples — the confidence intervals are wide enough to overlap.

The credibility spectrum

Not all benchmarks carry equal weight. Here is how to read them:

Martian's benchmark currently sits at the top of this spectrum — independent researchers, open methodology, dual offline-online approach. But even their offline component used only 50 PRs.

What no benchmark has tested yet

Every published benchmark tests tools — CodeRabbit, Qodo, Greptile, Augment. None of them test the underlying models directly.

This matters because tools add layers on top of base models: custom prompts, retrieval-augmented generation, multi-agent workflows, post-processing filters. When CodeRabbit scores 51.2% F1, you don't know whether that reflects Claude's capability, CodeRabbit's prompt engineering, or their post-processing pipeline.

Questions that remain unanswered:

• Claude vs Gemini vs GPT on identical code review prompts — no head-to-head model comparison exists for the code review task specifically
• False positive rates by model — which model produces the most noise?
• Performance across 10+ languages — every benchmark uses the same five repos
• Cost-adjusted accuracy — which model gives you the most bugs per dollar?
• Hallucination rates — how often do models reference APIs, functions, or variables that don't exist?

What we're building: an open code review benchmark

We are constructing a benchmark designed to fill these gaps. The full methodology is published at BENCHMARK-METHODOLOGY.md and summarized here.

10 repositories, 10 languages

We added Rust, C++, Kotlin, PHP, and Swift because current benchmarks only test Python, TypeScript, Go, Java, and Ruby. A model that excels on Python may struggle with Rust's ownership system or Swift's protocol-oriented patterns.

100 PRs with 150 injected bugs

Each repo contributes 10 PRs with 1-3 injected bugs per PR. Bug categories span five groups:

• Functional bugs (40%): off-by-one errors, null references, race conditions, resource leaks
• Security vulnerabilities (25%): mapped to CWE Top 25 — SQL injection, XSS, path traversal, SSRF, missing authorization
• Performance issues (15%): N+1 queries, unbounded collections, blocking calls in async contexts
• Code quality (15%): dead code, hardcoded secrets, missing input validation
• API misuse (5%): deprecated APIs, wrong argument ordering

Bug injection uses three methods: reversing real bug-fix commits (Greptile's approach), LLM-based synthetic injection with human validation (Qodo's approach), and reversing real CVE fixes (OWASP approach).

5 models, head-to-head

Every model receives the same system prompt, the same PR diff, and the same repository context. Temperature set to 0.1 for reproducibility. Each PR runs three times per model with majority vote scoring to reduce variance.

Scoring: LLM-as-judge with published rubric

Following Martian's approach, an LLM judge classifies each model comment against ground truth:

• Exact match (1.0): correct file, correct line range (±5 lines), correct bug category
• Partial match (0.5): correct file, general area (±20 lines), related category
• No match (0.0): wrong file or wrong location

We report precision, recall, F1, false positive rate, hallucination rate, cost per PR, and latency. All with 95% bootstrap confidence intervals.

Everything open-source

The test suite, prompts, ground truth, raw API responses, and scoring code will be published under Apache 2.0. Anyone can reproduce our results, challenge our scoring, or add new models and repositories.

How to read benchmark numbers without getting misled

When you encounter AI code review benchmark claims — ours included — apply these filters:

Check who ran the test. If the vendor tested themselves, expect optimistic numbers. Look for independent evaluations or at least published raw data that others have verified.

Ask for precision AND recall. A single "accuracy" or "catch rate" number hides the precision-recall tradeoff. A tool with 80% recall and 20% precision catches most bugs but drowns you in false positives. A tool with 90% precision and 30% recall is quiet but misses most issues.

Look at the sample size. Fifty PRs is borderline. One hundred is adequate. Claims based on "thousands of PRs" without published methodology deserve skepticism — large numbers don't help if the scoring is opaque.

Check if results are reproducible. Can you run the same benchmark on your own code? If the test suite isn't published, the results are assertions, not evidence.

What practical accuracy means for your team

The raw F1 numbers — 45%, 51%, 60% — sound low. Here is what they mean in practice:

A tool with 55% F1 and 60% precision on a PR with 5 real issues will typically find 3 of them and add 2 false comments. A senior developer spends thirty seconds dismissing the false positives and saves fifteen minutes catching three bugs they might have missed.

The question isn't whether AI code review is perfect. The question is whether catching 3 out of 5 bugs automatically — even with some noise — is worth more than catching 0 out of 5 because nobody had time for a thorough review.

For teams that review 20+ PRs per week, even 50% recall with reasonable precision saves hours of review time. The math works at current accuracy levels. It works better when you run multiple models and compare results.

The same tool scores 36% to 82% depending on who tests it

We spent two days pulling every published benchmark into a single spreadsheet, and the result genuinely surprised us. The same tool, evaluated by different organizations on different bug sets, produces scores that barely overlap:

DeepSource tested on 165 real CVEs from the OpenSSF dataset — security vulnerabilities, not code quality issues. That single methodological choice flipped the leaderboard. Cursor BugBot went from middle-of-the-pack to first place. CodeRabbit dropped from respectable to last.

The takeaway is not that any benchmark is wrong. The takeaway is that a tool's score tells you how it performs on that specific test, not how it performs on your code.

What AI code review consistently misses

CodeRabbit did something unusual for a vendor — they published data that makes the entire category look bad. Their team analyzed 470 pull requests, split between 320 AI-co-authored and 150 human-only, and the pattern they found cuts against every marketing page in this space: AI catches the bugs that matter least and struggles with the bugs that cost you the most.

Source: CodeRabbit State of AI vs Human Code Generation Report, 470 PRs

A separate academic study tested GPT-4o and Gemini 2.0 Flash on code review specifically. GPT-4o achieved 68.5% correctness when given problem descriptions, but generated harmful suggestions — code changes that make things worse — 10.4% of the time. That number jumped to 23.8% when the model reviewed code without context about what to look for.

The practical risk is not just missed bugs. It is that AI sometimes introduces new ones through its suggestions.

47% of developers now use AI code review — but 96% don't trust it

Adoption doubled every year between 2023 and 2025, which is remarkable given that accuracy barely moved in the same period. The numbers tell a story about developer pragmatism over perfectionism:

The scale numbers are staggering when you line them up. GitHub reported 60 million Copilot code reviews since April 2025 — one in five pull requests on the entire platform now gets AI feedback before a human looks at it. Microsoft's internal engineering team runs AI review on 90% of PRs across 5,000 repositories, roughly 600,000 reviews per month. CodeRabbit just closed a $60 million Series B after reviewing 13 million PRs across 2 million repos. These are not experimental pilots — this is infrastructure.

The trust gap is where it gets interesting. SonarSource surveyed 1,100 developers and found that 42% of committed code is now AI-generated — projected to reach 65% by 2027. But 96% of developers do not fully trust that AI-generated code works correctly, and 38% say reviewing AI code takes more effort than reviewing human code.

LinearB's 2026 engineering benchmarks report put a number on what most reviewers already feel in their gut: AI-authored PRs carry 1.7x more issues than human PRs (10.83 vs 6.45 per PR) and get accepted at barely a third the rate — 32.7% versus 84.4% for human code. Teams also wait 4.6x longer before picking up AI PRs, which suggests reviewers treat machine-generated changes with a level of suspicion that the accuracy data now justifies.

The paradox: more AI-generated code creates more need for AI-assisted review, because humans cannot keep up with the volume. But the AI doing the review misses the same categories of bugs that the AI writing the code introduces.

What a code review costs — AI vs human vs production

Nobody publishes this comparison, so we calculated it from current API pricing and industry salary data.

Based on: median PR = 5,000 input tokens + 800 output tokens (arxiv research reports 3,937 avg tokens/PR). API pricing from official pages as of April 2026.

Running all three models on every PR costs seven and a half cents. A hundred PRs per month costs $7.50 in API fees. The same volume of human review at mid-level rates costs $4,000.

The cost argument for AI review has never been about replacing human reviewers. It is about running a $0.075 first pass that catches 55-65% of issues before a human spends twenty minutes on the ones that remain. The human reviewer focuses on architecture, business logic, and concurrency — the exact categories where AI performs worst.

A bug caught in code review costs roughly $37 to fix (30 minutes of developer time). The same bug found in production costs $1,125 on average — a 30x multiplier documented across multiple engineering organizations. One prevented production bug per month produces a 15,000% return on $7.50 in monthly API costs.

Sources and benchmark data

All benchmark data in this article comes from primary published sources:

Benchmarks:

1. Martian Code Review Benchmark — withmartian.com/post/code-review-bench-v0 and GitHub
2. Greptile AI Code Review Benchmarks — greptile.com/benchmarks
3. Augment Code Review Benchmark — augmentcode.com/blog
4. Qodo AI Code Review Benchmark — qodo.ai/blog
5. CodeAnt AI Benchmark — codeant.ai/blogs
6. DeepSource OpenSSF CVE Benchmark — deepsource.com/blog/ai-code-review-benchmarks

Adoption and industry data: 7. GitHub Copilot: 60M Code Reviews — github.blog ✅ Verified 8. CodeRabbit Series B ($60M, 13M PRs) — coderabbit.ai/blog ✅ Verified 9. CodeRabbit AI vs Human Code Report — coderabbit.ai/blog ✅ Verified 10. SonarSource State of Code 2026 — sonarsource.com/blog ✅ Verified 11. LinearB 2026 Engineering Benchmarks — linearb.io/resources 12. JetBrains Developer Ecosystem 2025 — devecosystem-2025.jetbrains.com 13. Stack Overflow Developer Survey 2025 — survey.stackoverflow.co/2025/ai 14. Microsoft AI Code Reviews at Scale — devblogs.microsoft.com

Academic research: 15. Evaluating LLMs for Code Review (arxiv 2505.20206) — GPT-4o 68.5% accuracy, 10.4% harmful suggestion rate 16. Automated Code Review In Practice (arxiv 2412.18531) — avg 3,937 tokens per PR review