The Azure DevOps buyer question keeps landing in my inbox: what's the best AI code review tool for Azure Repos? Almost every reviewer in this space shipped GitHub-first, so I went and read the primary docs instead of the comparison listicles. What I found is less a quality question than an availability one. On Azure Repos, whether your AI reviewer runs at all is a config lifecycle you own, not a feature you install.
Here's the actual shape of it on two tools people ask about most.
Copilot code review on Azure Repos is a three-switch cascade
Microsoft's own prerequisites table for Copilot code review spreads setup across four roles before Copilot can comment on a single pull request:
- A Project Collection Administrator enables it at the organization level.
- A Project Administrator enables it at the project level.
- A Repository owner or administrator enables it per repo.
- Individual users opt in through Preview features, unless the admin turns the preview on for everyone.
This is the part people miss when they compare feature checkboxes. On GitHub, you install an app and it reviews. On Azure Repos, three separate people with three different permission levels have to agree, in order, before the reviewer exists. If your org has the classic split where repo admins are team leads but org settings live with a platform team, that's a ticket you file across org boundaries just to try a preview feature.
And it is a preview. The doc says it plainly: "This feature is in limited preview", preview capabilities "might change or be removed without notice", and preview features "have no Service Level Agreement (SLA) and limited support". TFVC isn't supported either, only Git repos.
Then there's billing. Copilot code review usage is billed through Azure Cost Management against an Azure subscription linked to the org, and the doc says higher "review effort" levels consume more tokens and can cost more. So the effort knob on a repository is a spend knob, set by an admin, with a project-level default and per-repo overrides that a project admin can allow or lock.
None of this is a knock on the model. It's just that availability and cost both live outside the PR, in three admin consoles, on a preview timeline.
CodeRabbit trades those admins for a PAT that quietly expires
CodeRabbit is the other name that comes up for ADO shops, and it does support Azure Repos. The setup is documented in CodeRabbit's Azure DevOps platform docs, which call out two permission kinds: standard repository and work-item access, plus permission to manage service hooks (webhooks).
There's no native OAuth app for the Azure DevOps integration. You connect with a Personal Access Token tied to an Azure DevOps user. A practitioner write-up on CodeRabbit + Azure DevOps setup notes spells out why that matters more than it sounds:
"When they do, CodeRabbit silently stops reviewing PRs. If the PAT belongs to an engineer who leaves the company, you lose code review across the org with no warning."
The recommended fix is a dedicated service account, Reader at org level, Contributor on the repos you want reviewed, with scopes for Code (Read & Write), Pull Request Threads (Read & Write), Project and Team (Read), and User Profile (Read), and a rotation date on a shared calendar.
Read that against the GitHub flow again. GitHub's OAuth app handles credential rotation transparently. On Azure Repos you own it. If you wire up the branch policy so CodeRabbit is a required status check and then the token lapses, you haven't just lost comments, you've possibly gated merges on a check that stopped reporting. Same write-up warns the status name can drift between CodeRabbit versions, so a required check can go stale after an update.
The thing both tools have in common
On Azure Repos, your AI reviewer is not a thing that is on. It's a thing you keep on. Copilot's version is three admin toggles and a preview flag. CodeRabbit's version is a service-account credential with an expiry date. Different machinery, same property: the review gate's liveness depends on configuration state that lives somewhere other than the diff.
That's why "does it review ADO PRs" is the wrong first question. The better one is "how do I know it's still reviewing." A reviewer that silently stopped three weeks ago produces exactly the same evidence as a reviewer that ran and found nothing: a clean pull request. This is the review-side version of what Cory Doctorow's glyph.im piece on what a serious AI product would look like argues about chatbots. If the product tells you it can be wrong and pushes checking onto you, but hands you no tool to check, you can't treat it as a tool. The same applies here, except the failure mode isn't a bad comment, it's no comment.
I wrote about the general problem of a reviewer that never fires under verify your AI reviewer actually follows your team's rules, and the same canary trick fixes the ADO liveness question.
A five-minute liveness check you can run on ADO
- Land a small commit that contains one obvious, unambiguous defect. Unused import, an off-by-one in a loop bound, an empty catch. Something a competent reviewer flags every time.
- Open it as a PR against the same target branch your real PRs use.
- Wait the amount of time you'd expect a review to land. If the tool's promise is "within 5 minutes", wait 10.
- If no comment arrives, you have a config problem, not a quality problem. Check the org toggle, the project toggle, the repo toggle, the preview opt-in, and the PAT expiry in that order.
- Once a comment does land, keep the canary in a private branch you re-run monthly, or after any credential rotation you don't control.
That last step matters because the failure is silent and slow. Nobody files a bug for the absence of a comment on a PR that looked fine.
Where self-hosted fits into this
If your real constraint is "we don't want a preview feature and we don't want someone else's PAT," the honest options shift. On GitHub, GitLab, or Bitbucket, Kodus is open source and self-hostable, and you bring your own model keys, so the availability of the reviewer is tied to your infrastructure and your rotation schedule rather than a vendor's preview flag or an org admin's toggle. That's the trade you're actually making: more of the lifecycle moves onto your team. Its current integrations are GitHub, GitLab, and Bitbucket, so it isn't the answer if you're staying on Azure Repos today, but it's worth naming in the same breath because it's the clean version of the "we own the liveness" posture.
For ADO right now, the comparable vendor option with a real ADO integration is CodeRabbit, and self-hosted there is an Enterprise-only path. If you need air-gapped or data-residency guarantees, that gates your shortlist fast.
What to measure before you pick
The comparison that helps you decide isn't the feature grid. It's these:
- Time to first review comment on a canary PR, measured by you, not the marketing page.
- Whether the check can be a required branch policy, and whether its status name survives a tool update.
- Who owns the credential, and whether the reviewer stops silently when it rotates.
- Comment density on a fixed two-week slice of real PRs before and after, so you can tell review volume from review noise.
That last one is where most AI review evals fall apart, because a reviewer that misses things and a reviewer that never ran look identical from the PR list. If you want the deeper version of that failure mode, my note on absence blindness in AI code review covers why recall is the number nobody publishes.
Pick the tool that makes it easy to prove it's running. On Azure Repos, that's the whole ballgame.
