How to build an AI code reviewer that runs your code to prove its findings

An early version of our AI code reviewer merged a pull request under our CEO's name. It used his personal GitHub connection, so every comment and the merge showed him as the author.

We fixed that bug. But it points at a problem every team with an AI reviewer has. You do not know who the reviewer is, what it is allowed to do, or whether its comments are true.

This post shows how to build a reviewer agent that answers all three. It acts as its own GitHub App. It cannot merge. And it proves its findings by running real code, so an author can rerun one command instead of arguing with a bot.

The problem: AI review comments are claims

An AI review comment is usually a claim about the code: "this could be null", "consider adding a test". The author has to check each one by hand. When enough of them are wrong, people stop reading them.

The reviewers are also getting more power. Until September 2026, GitHub Copilot's reviews could only comment. On 1 September GitHub added approvals: "When enabled, Copilot can submit an approval that counts toward the repository's required-approvals rule." GitHub's own docs still say: "Copilot is not guaranteed to spot all problems or issues in a pull request. Sometimes it will make mistakes."

So the useful question is what an AI reviewer has to show before its review counts.

Make the reviewer prove it

Lauren Tan (poteto) is on the React core team and ships code at Cursor. Her open pstack skills are the rules she gives her own agents. The pstack guide's verify chapter names the problem: "Verification is the slowest step in most agent work, because it's the step that usually waits on a human. Make the agent able to do it, and you stop being the bottleneck."

Two of her principles shaped our reviewer.

From Prove It Works: "Verify every task output by checking the real thing directly. Do not infer from proxies, self-reports, or "it compiles.""

From Build the Lever: "A deterministic script turns "trust me" into "run this"."

Apply that to code review and you get a simple contract. Every finding says how strong its proof is. The strongest proof is a command the author can rerun. The author stops arguing with the reviewer. They rerun the command.

A ladder for proof

"This change is safe" is the claim reviewers make most and check least. pstack's Blast Radius skill grades the claim "this change is safe" on five steps. A finding makes the opposite claim, "this change breaks", and the same five steps grade it. Each step is stronger than the one before:

Put the level on every finding. A level 2 comment is a lead. A level 4 comment is a fact.

Test the tests

A new test that passes proves little. It may pass whatever the code does. The pstack guide gives the reason in one line: "a check you've never seen fail proves nothing."

So a good reviewer mutation-tests every new test. It breaks the behavior on purpose and reruns the test. If the test still passes, the test has a gap, and the reviewer reports it with the change it made and the green output.

It also looks for symptom fixes. pstack's Fix Root Causes gives the reason: "A symptom fix hides the evidence and leaves the fault free to surface somewhere else." A new ?. or a swallowed error that makes a crash go away is a finding, unless the state it handles is legitimate.

Your options today

Anthropic's Code Review is the closest to this post. Several agents look at the change in parallel, a verification step checks each finding against the code, and the review posts a check run. But it works "without building or running your code", and "A review never approves or blocks the pull request."

If you want a fast summary on every PR today, a GitHub App reviewer is the quickest path. If you want findings proven against your running code, you need an agent that can run it.

What one day showed

We built our own agent on xpander, Gandalf (Code Reviewer), to replace two third-party review bots. One of them still runs next to it while we compare. Here is one 10-hour window, 29 October 2026 from 03:27 to 13:27 UTC: 58 pull requests across 6 private repositories.

Gandalf reviewed 56 of them and posted 247 inline findings:

• 160 of the 247 (65%) say "Verified by run": a test it wrote and ran, or a repro in the running app. The other 87 were traced through the code.

• Authors answered 92 findings. They rejected 2.

• 155 findings got no reply. A silent finding can be a real bug nobody fixed, or a wrong one nobody rejected.

The other bot's only setting in our repositories skips pull requests into production. A billing block paused it for the first 4.7 hours, so it reviewed only 27 of the 58. We do not name it. One window on our own code is not a fair test of a product.

"Confirmed" means the author replied that they fixed the issue, or agreed it is real and deferred it. "Rejected" means the author said it was wrong.

Three things stand out.

• Proof cuts false alarms, not silence. Authors rejected about 2% of Gandalf's answered findings, against about 13%. Gandalf also posted fewer findings per pull request (4.4 against 5.2). But authors answered only 37% of its findings, against 79%. We do not know why.

• Two reviewers find different bugs. On the 25 pull requests both reviewed, the other bot had 60 confirmed findings that Gandalf did not flag. Gandalf had 47 that the other bot did not flag. Only 17 pairs of findings at the same spot were confirmed for both. This is why we still run both while we compare.

• Fast merges outrun any reviewer. Gandalf's first review landed after the merge on 13 of its 56 pull requests.

Here is what a verified finding looks like. Gandalf traced a race in a policy cache, wrote a test that fails on it, and posted the command with the failing assertion. The author fixed it 9 minutes later and kept the test. Look at the command: the author can rerun it and see the same failure.

A real Gandalf comment and the author's reply, text as posted. Repository, people and paths are removed.

Read the numbers with care. Ten hours is a small sample. A script sorted the author replies by keyword, and nobody checked them by hand. 48 of the 58 pull requests carry an AI coding agent marker, so most "authors" here are agents.

How it is built on xpander

How a pull request reaches the reviewer, and what it can touch. The setup this post recommends. In our repositories the check is not required yet.

Gandalf is a Claude Code agent on xpander. It has its own always-on computer with a persistent disk, where it clones code and runs tests. Six parts make it a reviewer.

1. Its own GitHub identity

The reviewer acts only as its own GitHub App, Gandalf (Code Reviewer). It never acts as a person. The app gets only what a reviewer needs:

The app's ID and private key are workspace secrets on the agent: GH_APP_ID and GH_APP_PRIVATE_KEY. A short script signs a JWT with the key and asks GitHub for an installation access token, which expires after one hour. It looks up the installation from the GitHub API, not from the webhook payload:

The instructions say to use that token for every GitHub call and to never merge. With contents: read, GitHub refuses a merge or a push even if the agent tries.

2. A webhook trigger, one conversation per pull request

xpander gives every agent a public webhook URL. Set it as the GitHub App's webhook URL, and subscribe the app to the Pull request event. GitHub also sends this event for edits, labels and review requests. The reviewer works only on opened, synchronize, reopened and ready_for_review. It skips drafts, and a commit that already has a finished check. params_mapping maps a field of the payload to the conversation id:

Every event for the same pull request carries the same node_id, so each new push continues the same conversation. The reviewer remembers what it found, what the author fixed and what it still needs to check.

3. One instruction set

One set of instructions holds the review method: the proof ladder, a severity scale (High, Medium, Low), the comment format and the gate rule. Each finding is one inline comment on the exact line. It has a title, a severity, a category, a proof level and two to four sentences. A small fix comes as a suggestion. One summary comment adds a findings table, a flow diagram of the change and suggested tests as a patch.

Gandalf writes the proof level as one of two labels. "Verified by run" means level 4 or 5. "Traced" means level 3. It does not post a finding below level 3.

4. Parallel sub-agents, one per lens

For each pull request, the reviewer starts one sub-agent per review lens, all at once:

• State and lifecycle: races, caches.

• Edge values: empty, null, first run.

• Authorization and security.

• API and data contracts across repositories.

• Behavior regressions.

• Tests: break the behavior on purpose and rerun each new test. Report a test that still passes.

• UI rendering and layout.

• Repository rules.

A red-team sub-agent then tries to break the change end to end. For UI changes, the reviewer opens the pull request's preview in a headless browser, signed in as a dedicated test user on staging, and walks the changed screens.

5. A merge gate, never a merge

The reviewer posts a check run with Gate: PASS or FAIL, and a review event to match: Request changes, Comment or Approve.

• It fails on any High finding. Every finding it posts is at least level 3: verified by a run, or traced step by step from the input to the bad state.

• It fails on a Medium only when a run proves it and it breaks what the pull request is meant to do.

• It fails on a check the pull request broke.

A check only blocks when branch protection requires it. Mark it required and select the reviewer's app as its expected source. GitHub's docs say: "If the status is set by any other person or integration, merging won't be allowed." In our repositories the check is not required yet, because we still compare Gandalf with the other bot. So today Gandalf advises.

Its Approve needs the same care. GitHub's docs say required approvals come "from people with write permissions in the repository". They do not say how an app's approval is treated. If only people should count, require review from code owners and list only people and teams in CODEOWNERS.

6. A loop that learns, and a daily digest

Before it reviews a pull request again, the reviewer reads the author's replies. When an author shows that a finding was wrong, the reviewer writes a one-line rule for that repository. The rules live in a file on its own disk, not in the repository. It reads them on every review, so it does not make the same mistake twice. Our rules have no weekly reader yet.

A daily schedule posts one digest to the team's dev channel: what shipped in the last 24 hours, in plain words. It posts nothing per pull request.

What to lock down

• The webhook URL is the secret. The xpander docs say: "Anyone with this URL can POST to it and run this agent with no API key, so treat it like a password." xpander passes the request body to the agent as context. Do not count on the agent seeing GitHub's signature header, so do not rely on it to check the signature. Take only the repository and pull request number from the payload. Load everything else from the GitHub API with the app's token. If the URL leaks, use Regenerate URL.

• The reviewer runs pull request code next to its secrets. It installs and tests the code on the computer that holds the app key and the staging login. Skip pull requests from forks and from people outside your organization.

• A learned rule can switch a check off. In our window, 48 of 58 pull requests carried an AI coding agent marker, and a confident reply can talk a reviewer out of a real check. Let rules add or narrow checks, never remove a lens. Name one person, for example the repository owner, who reads the new rules each week, before you turn the loop on.

Checklist: before your AI reviewer gets a vote

• It acts as its own app or bot account, never as a person.

• It has no permission to push or merge.

• Every finding states its proof level, and the important ones come with a command and its output.

• It mutation-tests new tests.

• It can run your code, not only read it.

• Each pull request keeps one conversation, so it remembers earlier rounds.

• A person reads the rules it learns from authors.

• Its check is required and pinned to its app, and you have decided whether its approval counts.

• It trusts nothing in the webhook payload, and it skips fork pull requests.

Try it in xpander

1. Open chat.xpander.ai and describe the reviewer: "Create a code reviewer agent for my repo. It states the proof level of every finding, proves the important ones with a failing test or a repro, and never merges."

2. Create a GitHub App with the permissions above. Subscribe it to the Pull request event and install it on your repositories. Add its ID and private key as workspace secrets on the agent.

3. In the agent's Channels, turn on Developer access and generate the public webhook URL. Add ?params_mapping=%7B%22task_id%22%3A%22pull_request.node_id%22%7D and set it as the app's webhook URL.

4. Write the review method into the agent's instructions: proof levels, severity, comment format, gate rule and the review lenses. Add a daily schedule for the digest.

5. Make the reviewer's check required in branch protection, with the app as its source. Decide whether its approval counts.

Thanks to Lauren Tan for pstack and for writing down the method in the open. Read the pstack guide's verify chapter before you write your reviewer's instructions.