Rolling out AI code review without getting burned: the checklist

• It can comment and request changes. If it can approve, an approval is a posted review, not a merge. Branch protection and your existing gates still apply.
• You have written an always-human list. At minimum: authentication and authorization, secrets and encryption, payments and pricing, infrastructure and CI configuration, and anything that changes who is allowed to approve.
• The file that defines what is sensitive is protected by code owners and read from your main branch, so a pull request can’t edit the rule that judges it.
• The reviewer has the fewest permissions possible: read the repository, post comments. Nothing that can fetch internal documents, trigger pipelines or call other systems.

Rewind’s version of this is “keeping the judgment (AI) and the gate (deterministic code) separate.” The model gives findings; a small, tested piece of ordinary code turns them into a decision.

• Descriptions, comments, commit messages, file contents, images and repository instruction files are all treated as data, never as orders.
• The reviewer’s output can’t trigger anything on its own: no merges, no deploys, no posting of content it fetched elsewhere.
• You have tested it once: open a throwaway pull request with an instruction hidden in the description and see what your reviewer does.

• It reviews alongside your human reviewers for at least a week without approving anything.
• You measure agreement on the pull requests both of them reviewed.
• You set a bar before going live. Rewind’s was 90% agreement; it reported about 99%.
• Every disagreement is read. Each one becomes a rule to fix or a category to keep human.
• You start with the boring categories: dependency bumps, docs and copy. Rewind’s early wins were mechanical version bumps.

• You know what “good” looks like. Intercom reports that over 19% of pull requests on its two main codebases are auto-approved, so roughly one in five. Rewind reported about 40% in early, small-sample data that it expects to settle lower.
• Every pull request that shipped without a human is labelled and logged with its evidence. That record is what an auditor asks for, and what you reach for when something breaks.
• Any engineer can send any pull request back to a person, at any time.
• Someone reviews the reviewer weekly. Changes to its prompts or policy go through ordinary pull requests; it never edits its own rules. Rewind expires suppressions after 90 days unless someone re-justifies them.
• Compliance is in the room early. Rewind brought its GRC lead in at the start; Intercom engaged its auditor before scaling.