PR Lens

Comparisons

PR Lens scores #1 on Martian's Code Review Bench

We ran Martian's offline Code Review Bench against our production reviewer six times. Our best run scored F2 .660, first on the board's default view. Every run and every comment is public

We forked the 50 pull requests in Martian's Code Review Bench into a public GitHub org, let our production reviewer loose on them, and scored the comments with Martian's own pipeline and judge. Then we did it five more times, because one run of any AI reviewer is partly luck.

Our best run scored F2 .660. On the board's default view, that is first of 22 tools.

Martian Code Review Bench, default viewF2 score, higher is better
  1. PR Lensby Coldtea.660
  2. Qodo Extended.651
  3. Cubic.649
  4. Augment.640
  5. Qodo.606
  6. GitHub Copilot.580
  7. Gitar.536
  8. Gemini.528
  9. GitLab Duo.527
  10. Sourcery.515
  11. Greptile.512
  12. CodeRabbit.511

Top 12 of 22 tools. Board scores from Martian, Opus 4.5 judge, Core set.

The bench counts what a reviewer catches and what it gets wrong

Martian's bench is 50 real pull requests from Sentry, Grafana, Keycloak, Discourse and Cal.com, with 158 known defects between them. A model judge reads every comment a tool leaves and decides which defects it caught (hits) and which comments are about nothing real (wrong comments).

Precision is the share of comments that were right. Recall is the share of defects that were caught. F1 weighs the two equally; F2, the board's default, counts a missed bug as worse than a noisy comment.

Across three runs, PR Lens stays in the top three

The same reviewer on the same 50 pull requests does not leave the same comments twice. Our three runs with the production setup landed at F2 .633, .660 and .644, a spread of almost three points.

Hits of 158Wrong commentsF1F2
PR Lens, best run10559.652.660
PR Lens, average of 3 runs102.358.3.642.645
Qodo Extended v210250.658.651
Cubic v210465.636.649
Augment10370.622.640

The average of those three is second on F1 and third on F2, a hair behind Qodo and Cubic. Two of the three runs were second on F1 on their own.

The verifier never found a bug, and it still mattered

Before PR Lens posts a comment that says how a language or a library behaves, a verifier tries the claim out, on the version the repository actually uses, and holds the comment back when the claim does not hold up.

To see what that is worth, we ran the bench three more times with the verifier switched off.

Hits of 158Wrong commentsF1F2
Verifier on (3 runs)102.358.3.642.645
Verifier off (3 runs)102.069.3.619.635

The verifier never found a single extra bug. It just stopped us posting eleven wrong comments a run. That is enough to move the F1 average from fourth to second, which I did not expect from a step that, on its own, catches nothing.

It isn't free. Across the first two runs it held back nine comments, and four of those were right: the same real bug, a cache key built with Python's per-process hash(), was withheld twice. A verifier that runs code can still run the wrong code, and get a confident wrong answer. We found one of those in production the same week, where the check ran on an older Python than the repository pins and confirmed a syntax error that isn't one.

What it catches every time

Some defects PR Lens found in all three runs. In Cal.com, a permission check required a user to be both team admin and team owner, where either should do: one && that should have been ||. In Discourse, a blocked email only stayed blocked if the next sign-up typed it in the same case, so User@Example.com walked past a block on user@example.com. In Keycloak, a constructor null-checked grantType twice and never checked rawTokenId.

None of these are clever. They are the kind of thing a tired reviewer reads straight past at the end of a 400-line diff, which is exactly when you want a second pair of eyes that doesn't get tired.

Some take more than a careful read. In Sentry, a new paginator handed a negative offset straight to a Django queryset slice, which Django refuses, so paging backwards through audit logs would throw. It also called math.floor on a key that, for that endpoint, is a datetime. In Grafana, a lock moved out from around an index build, so two requests for the same key could both build the same expensive index at once. In Cal.com, a cancellation ran its calendar deletions and payment refunds inside forEach with an async callback, so nothing waited for them and a failure went nowhere.

These are the bugs that pass review because the code looks right. Each one only shows up when you know how the thing it calls behaves, and PR Lens caught all three, every run.

Where we occasionally lose

Qodo Extended v2 leaves fewer wrong comments than we do: 50 a run against our 58. That is the whole gap on F1, and it is the next thing we are working on.

Most of our wrong comments are not about language or library behaviour, so the verifier never sees them. They are claims about the repository itself: a value that can't be null, a caller that doesn't exist, a setting nobody uses. Those are harder to check than a line of Python, and they are where the next few points are.

Every result is public, down to each comment

Each run opened fresh pull requests on the forks, so every comment PR Lens left is still there to read at github.com/pr-lens-bench. The judge's verdict on every comment, for all six runs, is in pr-lens-bench/results, with the table above worked out from it.

If you think the judge got one of ours wrong, open the comment and its verdict side by side. That is how we went through our own misses, and it is how we found the verifier bug above.

Method

Six runs on 16 and 17 November 2026, each on fresh pull requests opened on public forks of the 50 benchmark pull requests and reviewed by PR Lens in production, with the settings customers had at the time. Runs 1, 2 and 6 had the verifier on; runs 3, 4 and 5 had it off. Scoring used Martian's pipeline at commit e616e84 (extract, dedup and judge), Claude Opus 4.5 as the judge, and the Core set of 158 defects, compared against the board's published scores under the same judge. Precision and recall come from the judge's per-comment verdicts, so a single run moves by a few points either way; the averages are the numbers to lean on.

Questions people ask

Is PR Lens on Martian's leaderboard?

Not yet. Martian runs the pipeline itself for every tool it lists, and we have asked to be added. These numbers are from our own runs of their pipeline, with their judge, on their 50 pull requests. The judge files for every run are public so anyone can check them.

Why lead with F2 and not F1?

F2 is the board's default view. It counts a missed bug as worse than a wrong comment, which matches how most teams feel about review. On F1 our best run is second, .006 behind Qodo Extended v2.

Can I check these numbers myself?

Yes. Every comment PR Lens left is on the public forks at github.com/pr-lens-bench, the judge's verdict on each one is in pr-lens-bench/results, and Martian's pipeline is open source, so you can rerun the scoring and get the same table.

Sources

Keep reading

Turn it on

Add PR Lens to GitHub, GitLab or Bitbucket, and every new pull request is explained and reviewed. Or paste the prompt into the coding agent you already use