AI code reviewers are everywhere now. Open a pull request, and a bot jumps in with comments: a possible bug here, a missing test there, a nitpick about naming.

But here’s the awkward question nobody could answer well: which of these AI reviewers is actually good?

And if you’re building your own code review tool, there’s an even harder question: how do you test that it catches real issues, avoids noisy false positives, and actually improves with each change?

GitHub’s answer, announced on October 5, 2026, is ReviewBench, an open benchmark that scores AI code reviewers on real-world pull requests. Here’s what it is, why it matters, and how it works, in plain English.

Table of contents

• The 30-second version
• First, a few quick terms
• The “Why”: AI reviewers are hard to compare
• The “What”: A benchmark built on real code
• The “How”: Four ideas that make it work
  • 1. An answer key built from many sources
  • 2. Credit for finding things nobody else found
  • 3. Scores you can tune to your taste
  • 4. Checked by humans, versioned for reproducibility
• Does it actually predict real-world results?
• Want to try it? Here’s how
• The bottom line

The 30-second version

• What: A public test that measures how well AI code reviewers find real problems in code.
• Why: Until now, there was no reliable, apples-to-apples way to compare them.
• How: Run every reviewer on the same 219 real pull requests, then score their findings against a carefully validated answer key.

First, a few quick terms

If you’re new to this space, these five words will carry you through the rest of the post:

• Pull request (PR): A proposed change to a codebase that teammates review before it gets merged.
• Finding: A single issue the reviewer flags, like “this line can crash if the list is empty.”
• Golden set: The answer key, meaning a vetted list of the real issues in each PR.
• Precision: Of everything the reviewer flagged, how much was actually valid? High precision means low noise.
• Recall: Of all the real issues, how many did the reviewer catch? High recall means fewer misses.

Think of it like a smoke detector. Precision is “does it only go off when there’s actually a fire?” Recall is “does it go off every time there is a fire?”

The “Why”: AI reviewers are hard to compare

Not all AI reviewers behave the same way. Some flag tons of issues but bury you in noise. Others stay quiet and only speak up for serious bugs. Some are great at security, others at catching small maintainability problems.

None of these is automatically “best.” It depends on what your team wants. And that’s the problem: without a shared, trustworthy test, choosing a reviewer is mostly guesswork.

GitHub says existing benchmarks usually compromised somewhere, whether on the quality of their labels, how much ground they covered, or how closely they resembled real-world review work. ReviewBench is their attempt to get all three right at once.

There’s a second motivation too. Teams building AI reviewers need a quick offline test that predicts whether a change will actually make things better for real users, before they roll it out.

The “What”: A benchmark built on real code

ReviewBench isn’t a set of toy examples. The team studied 103.9 million GitHub pull requests to understand what real review work looks like, then built a test set to match.

The result:

• 219 pull requests
• from 187 public open-source repositories
• across 19 programming languages

The mix of languages and repository sizes mirrors GitHub as a whole. The one intentional tweak: they leaned away from tiny one-file changes and toward meatier, multi-file PRs, since that’s where good reviewing really matters.

Every finding in the answer key is tagged by:

• Severity: Critical, Medium, or Low
• Category: Correctness, Security, Reliability, Maintainability, Testing, and more

That tagging is what lets you slice results to fit your priorities (more on that below).

The “How”: Four ideas that make it work

1. An answer key built from many sources

No single reviewer, human or AI, catches everything. So ReviewBench builds its golden set from a mix of sources:

• Comments from real human reviewers
• Issues revealed by the author’s follow-up fixes
• Static analysis tools
• Several frontier AI models from different model families

Duplicate findings get merged, so ten sources spotting the same bug doesn’t count as ten bugs. Then every finding is checked against one shared rubric: to count, it must be true, relevant, and non-trivial. Where a finding came from doesn’t earn it a pass.

The grading is done by an AI judge (Claude Sonnet 5), and GitHub publishes both the rubric and the judge setup so anyone can reproduce the results.

2. Credit for finding things nobody else found

Here’s a clever twist. What happens when a reviewer spots a real bug that isn’t in the answer key? In most benchmarks, it gets marked wrong. That’s unfair, especially as AI reviewers get smarter.

ReviewBench reports two families of scores:

• Grounded metrics: Strict scoring against the known answer key. This is the fair, side-by-side comparison, and grounded recall is the headline number.
• Augmented metrics: The judge also evaluates findings outside the answer key and gives credit for valid new discoveries. Useful as a deeper per-system diagnostic.

3. Scores you can tune to your taste

Want a reviewer that only flags critical bugs? Filter by severity. Care mostly about security? Filter by category.

You can also adjust the balance between precision and recall using something called the Fβ score. Turn the dial one way if you want maximum coverage, the other way if you hate noisy comments. The leaderboard re-ranks to match your preference.

4. Checked by humans, versioned for reproducibility

Before launch, senior engineers who weren’t involved in building the dataset re-labeled every answer-key finding from scratch. They agreed with ReviewBench 96.6% of the time.

Every evaluation also pins the exact versions of the dataset, judge, and matching logic, so scores stay comparable over time. GitHub also publishes its known limitations, which is a nice touch of honesty.

Does it actually predict real-world results?

This is the part that makes ReviewBench more than an academic exercise.

GitHub used it to test changes to Copilot code review before shipping them. In one experiment, they combined several independent model runs into a single review (an “ensemble”). ReviewBench predicted better precision, better recall, more comments, and lower cost.

Then they ran a live A/B test with real developers. Compared with the existing version, the results moved the same way:

Even better, ReviewBench predicted a 227% jump in critical comments. The real-world number? 262%. Same direction, similar magnitude.

GitHub’s takeaway: live experiments are still the final word, but ReviewBench is a fast, reliable early signal for which ideas are worth testing.

Want to try it? Here’s how

ReviewBench is available now as a research preview at review-bench.ai. You can:

• Browse the full dataset: every PR, finding, label, and tag.
• Compare reviewers on a public leaderboard.
• Test your own AI reviewer and improve it over time.

Submitting your own reviewer takes five steps:

1. Sign in with GitHub.
2. Register your agent with a container image, your config, and your own model key. ReviewBench supplies the judge.
3. Practice on a smaller 25-PR test set and tune as you go.
4. Run the real thing: all 219 PRs, three rounds, scored by the same judge as everyone else.
5. Get on the leaderboard: a maintainer reviews your submission first, and a score is published only if it’s your first entry or beats your previous best.

The bottom line

AI code review is quickly becoming a normal part of shipping software, but “trust me, it’s good” isn’t a strategy. ReviewBench gives developers a way to compare reviewers on real code, tuned to what they care about, and gives builders a test that actually tracks real-world impact.

If you use an AI reviewer, it’s worth a look at the leaderboard. If you build one, it’s worth running the benchmark.