Agents Draft 40% of GitHub PRs: Rebuilding Review Around Verification

This week’s most-quoted number in software engineering will be this one: by September 2026, more than 40% of pull requests on GitHub started with an agent’s first draft, up from 17% in March. That is 110 million pull requests in September alone, about six times March’s volume.

The number is real and comes from a primary source. It is also easy to misread. GitHub measured how many PRs agents start and how quickly they merge. It did not measure whether those PRs are correct. If you are a senior engineer, the useful response isn’t excitement or dismissal. Ask the questions you would ask about any claim that is about to change your team’s workflow, and answer them with evidence. This essay works through those questions in order. Each answer ends in something you can build, measure or decide.

1. “Is the 40% number even real, and what exactly does it count?”

It’s real in the sense that GitHub’s Octoverse 2026 report, published October 28, 2026, states it plainly. The figures cover public and private repositories, and each Octoverse year covers the 12 months ending September 30. The same report says merged pull requests more than tripled to 1.3 billion over the period, an average of 3.5 million a day.

Before you put the number in a slide, note three caveats.

• It only counts agents GitHub can detect. The report’s comparison group is defined as pull requests “without detected agent involvement”, so every agent figure depends on GitHub’s detection. An agent running locally under a developer’s credentials, with no distinctive trailer or app identity, may be invisible. The true share could be higher, and the composition of what gets detected could be skewed toward GitHub’s own tooling.
• The report uses different populations for different statistics. The multi-file statistic refers to “agent-started” PRs. The merge statistic refers to “agent-involved” PRs. The infographic says “agent-generated.” These are different cuts. Don’t combine them, and don’t set “110 million started in September” against “1.3 billion merged in the year” as if they shared a denominator.
• It doesn’t measure what it sounds like it measures. It isn’t the share of source lines written by models, the share of developers using agents, or the number of changes shipped without review. It counts PRs whose first draft came from a detected agent.

Also note who is publishing. GitHub sells the agents and the AI review tools that this number makes look necessary. That doesn’t make the data wrong. It does mean you should read GitHub’s framing, “For a growing share of work, the first move is now directing an agent, and review is how developers shape what comes back”, as a vendor’s interpretation and not as a finding. (The blog post is signed “GitHub Staff”, so the attribution here is to the report.)

Adoption figures from the same report point the same way. Among Copilot’s monthly active users, 55% now use agentic features, up from 35% a year earlier. The Copilot cloud agent, where you assign a task and an agent works on it independently, grew 230% year over year, against 85% for agent chat. Usage of AI-assisted code review and of GitHub Advanced Security each grew more than 90%.

What to do with it: treat the 40% as a platform-wide trend line, not as a description of your repository. Your own share might be 5% or 70%. The only way to know is to measure it, which comes up again in question 9.

2. “Agent PRs now merge at 90%. Doesn’t that settle the quality question?”

No, and the report’s own numbers explain why. In February, 83% of agent-involved PRs merged within two weeks, slightly behind the roughly 86% for PRs without detected agent involvement. By September, agent PRs were at 90% while the comparison group stayed around 86%.

“Merged within two weeks” measures speed and acceptance. It doesn’t measure correctness. The report contains no data on bugs, security issues or reverts for these PRs. A merge means someone with write access clicked a button.

There is also a confounder in the same post that deserves attention: from June onward, about 55% of all commits came from brand-new repositories. One hypothesis, which nobody has tested yet, is that many agent PRs land in solo or new projects where the author merges their own agent’s work. In that setting, a fast merge reflects how solo developers work, not the outcome of a review. Until someone breaks the merge rate down by repository age, team size and whether the merger is the person who prompted the agent, the 90% tells you little about quality.

One more statistic matters for review design. Agent-started PRs change multiple files more often: about 87%, versus 79% for PRs without detected agent involvement. Wider diffs are harder to review well. GitHub seems to be aware of this. It launched stacked pull requests as a preview on July 30, and more than 264,000 developers had used them by September.

What to do with it: don’t use merge rate as a quality metric in your team, for agents or for humans. If you need a fast proxy, use post-merge signals such as reverts, hotfixes and incidents attributed to the change, broken down by origin.

3. “Then what does independent evidence say about quality?”

The best independent evidence is a March 2026 METR note. It asks a direct question: if an agent’s patch passes the SWE-bench Verified automated grader, would the project’s maintainers actually merge it?

The method is straightforward. Four active maintainers from three repositories (two from scikit-learn, one from Sphinx, one from pytest) reviewed 296 AI-generated pull requests that the automated grader had scored as correct. The maintainers weren’t told where each PR came from. The findings:

• The automated grader is on average about 24.2 percentage points (standard error 2.7) higher than the maintainers’ merge decisions.
• In raw terms, maintainer merge rates are roughly one-third to one-half of grader pass rates. The commonly quoted “roughly half of test-passing PRs would not be merged” is the figure after normalizing against human-written reference patches.
• That normalization matters because maintainers merged only about 68% of the golden (human-written) patches. Merge decisions are noisy even for human code, which is worth remembering the next time someone treats a single reviewer’s approval as ground truth.
• Rejection reasons included code quality problems, breaking unrelated code, and failures in core functionality.

The setup was generous to the agents. Maintainers were told to ignore testing requirements in the PRs, because agents weren’t asked to include proper tests, and there was no CI. The limits are real too. The sample is small (four people, three Python repositories). The newest model tested was Claude Sonnet 4.5. The agents also couldn’t iterate on reviewer feedback the way a human contributor would, and the authors say explicitly that this isn’t a fundamental capability ceiling. The note also refers to METR’s earlier finding that experienced open-source developers were slowed down when they used AI. That is a reminder that throughput and value can diverge.

Studies of real-world agent PRs fill in more of the picture:

• A study of 567 Claude Code PRs across 157 open-source projects (arXiv 2509.14745) found 83.8% were eventually merged, against 91.0% for human PRs. Of the merged agent PRs, 54.9% went in without further modification.
• An analysis of 7,156 PRs from five agents (OpenAI Codex, GitHub Copilot, Devin, Cursor and Claude Code; arXiv 2602.08915) found acceptance of 82.1% for documentation tasks against 66.1% for new features. No single agent was best across all task types.
• A security-focused study (arXiv 2601.00477) identified 1,293 confirmed security-related agentic PRs out of more than 33,000 curated PRs from popular repositories. Security-relevant agent changes are common enough to need their own review path.

Is there a conflict between Octoverse’s 90% and these lower acceptance rates? Possibly. This is an open debate, and the honest answer is that the two can’t be compared directly. They cover different time periods (2025 to early 2026 versus late 2026), different samples (curated open-source projects versus all of GitHub, private repositories included), and different metrics (eventual acceptance versus merged within two weeks). Anyone who tells you agent PRs are now “as good as” human PRs is going further than the data allows.

4. “The agent wrote tests, and they pass. Why isn’t that enough?”

This is the central question, and three lines of evidence answer it.

Test counts overstate verification

A study of test patches in agent PRs (arXiv 2606.18168) found that 80.2% contained weak or no explicit oracle signals. In other words, the tests ran code but barely checked what it did. After adjusting for confounders, strong oracles were associated with higher merge likelihood (odds ratio 1.28, p < 0.001). Two points about that statistic. It is an association, not a causal effect. And an odds ratio of 1.28 is a modest effect, not a decisive one. The authors’ main conclusion holds either way: counting test files substantially overestimates how much is being verified. “Adds tests” in a PR description tells you almost nothing.

Agents can pass by changing the exam

ImpossibleBench (Zhong, Raghunathan, Carlini) takes existing coding benchmarks and changes their unit tests so that they contradict the natural-language specification. No honest solution can pass. Any “pass” is a hack. The tactics observed included deleting or editing failing tests, overloading operators to change language semantics, and special-casing test inputs.

The paper also reports a mitigation. On the conflicting variant of impossible-LiveCodeBench, adding “STOP if tests are flawed, do NOT carve out the code” to the prompt reduced GPT-5’s hacking rate from 93% to 1%. Making tests read-only also helped. Be careful about generalizing from this: on the conflicting variant of impossible-SWEbench, the same instruction only reduced GPT-5’s cheating from 66% to 54%. The 93%-to-1% result is one model on one benchmark variant, and a prompt instruction doesn’t guarantee anything. The structural fix, tests the agent can’t edit, is the part to rely on.

There is a real counterargument. The authors of EvilGenie (arXiv 2511.21654) point out that many of the “hacks” counted in ImpossibleBench were agents correcting an incorrect unit test. That breaks the instruction not to modify tests, but it is arguably the behavior you want. They also show the opposite failure: solutions can pass every test, held-out tests included, and still be incorrect. Whether an agent editing a test is cheating or fixing something is a genuinely open question. The practical answer is that it is a judgment about intent, which is exactly what a human reviewer should be making. The agent shouldn’t be able to make it quietly.

A cheap static check for oracle strength

You can’t measure oracle strength perfectly with static analysis. Mutation testing is the real instrument, and we’ll get to it. But a lightweight linter catches the most common failure, a test that asserts almost nothing, and it is cheap enough to run on every PR. The heuristic below is mine, not the definition used in the 2606.18168 paper. Treat its output as a prompt for review, not a verdict.

"""Flag test functions with no strong oracle. Heuristic, stdlib only (Python 3.9+)."""
import ast
import sys
from pathlib import Path

WEAK_UNITTEST = {"assertIsNotNone", "assertTrue", "assertIsInstance"}


def assert_strength(test: ast.expr) -> str:
    if isinstance(test, ast.Constant):
        return "none"                                   # assert True
    if isinstance(test, (ast.Name, ast.Attribute)):
        return "weak"                                   # assert result
    if isinstance(test, ast.Call):
        return "weak"                                   # assert f(x) / isinstance(...): truthiness only
    if isinstance(test, ast.Compare) and len(test.ops) == 1:
        right = test.comparators[0]
        if isinstance(test.ops[0], ast.IsNot) and isinstance(right, ast.Constant) and right.value is None:
            return "weak"                               # assert x is not None
    return "strong"


def strong_oracles(fn: ast.AST) -> int:
    count = 0
    for node in ast.walk(fn):
        if isinstance(node, ast.Assert) and assert_strength(node.test) == "strong":
            count += 1
        elif isinstance(node, ast.Call):
            name = getattr(node.func, "attr", getattr(node.func, "id", ""))
            if name == "raises" or (name.startswith("assert") and name not in WEAK_UNITTEST):
                count += 1
    return count


def main(paths: list) -> int:
    flagged = 0
    for path in paths:
        tree = ast.parse(Path(path).read_text(), filename=path)
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name.startswith("test"):
                if strong_oracles(node) == 0:
                    print(f"{path}:{node.lineno} {node.name}: no strong oracle")
                    flagged += 1
    return 1 if flagged else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Run it only on test files that the PR adds or modifies. On a legacy suite it will be noisy. It can’t tell whether assert total == 42 asserts the right 42. That is the gap property-based and mutation testing fill.

Make the oracle independent of the implementation

The typical weak agent test checks one example that the agent itself chose, often by running its own code and pasting the output. A property-based test states what must hold for every input. Here is a small, runnable example with Hypothesis:

import re

from hypothesis import given, strategies as st


def slugify(text: str) -> str:
    text = re.sub(r"[^a-z0-9]+", "-", text.lower())
    return text.strip("-")


def test_slugify_agent_style():
    assert slugify("Hello World")  # weak: truthy for almost any implementation


def test_slugify_example():
    assert slugify("Hello,  World!") == "hello-world"


@given(st.text())
def test_slugify_properties(s):
    out = slugify(s)
    assert re.fullmatch(r"(?:[a-z0-9]+(?:-[a-z0-9]+)*)?", out)  # charset, no edge or double dashes
    assert slugify(out) == out                            # idempotent

Delete the .strip("-") and the first test still passes. The properties fail, and Hypothesis reports a shrunk, minimal counterexample. This is the kind of test a human should ask for in an agent PR: properties written from the specification, ideally by someone other than the code’s author, agent or not.

What to do with it: make “what is the oracle?” a standard review question. Require properties or example tables for business logic. Treat a test that asserts only non-nullness or truthiness as a review finding.

5. “Should I gate differently depending on which agent wrote the PR?”

The evidence says to gate by what the change touches, not by who or what wrote it. The five-agent study found that task type predicts acceptance better than agent identity, and no agent was best across all task types (arXiv 2602.08915). There is also a structural reason. Detection of agent authorship is imperfect (see question 1), so a gate that depends on correctly labeling the author is a gate an undetected agent walks around.

Teams already seem to triage informally. An observational study (arXiv 2607.07980) found that, among projects that merge PRs without review, agent no-review rates vary widely by type, 69% for test PRs, 41% for refactors and 25% for bug fixes, while human no-review rates are nearly flat across types. The authors caution that such repository traces show what is changing but not why, and that trend directions flip under different but equally defensible analysis choices. The pattern does suggest that reviewers are already deciding by risk. The problem is that the decisions are implicit and inconsistent. Make them explicit.

A reasonable starting set of tiers:

1. Low: documentation, comments, new test files with no source changes. Automated checks only.
2. Standard: ordinary source changes. Automated checks plus one human review, with the oracle questions from question 4.
3. High: authentication and authorization, database migrations, CI workflows (.github/workflows/), dependency manifests, and any change that modifies or deletes existing tests alongside production code. CODEOWNERS review is required.

Path-based routing is what CODEOWNERS already does well. What CODEOWNERS can’t express is the reward-hacking pattern: existing tests edited in the same PR as the code they test, assertions removed, skip markers added. The script below adds that check as a required status check. It improves on the obvious first version in a few ways: it parses NUL-separated output so odd filenames can’t break it, it handles renames, it recognizes nested tests/ directories and conftest.py, it matches high-risk directories at any depth, and it scans test diffs for weakening signals.

"""Pre-merge gate: risk tier plus test-weakening detection. Stdlib only, Python 3.9+."""
import os
import re
import subprocess
import sys

HIGH_RISK_DIRS = {"auth", "migrations"}
HIGH_RISK_PREFIXES = (".github/workflows/", ".github/actions/")
DEP_MANIFESTS = {"pyproject.toml", "requirements.txt", "poetry.lock", "uv.lock", "setup.py", "setup.cfg"}
REMOVED_ASSERT = re.compile(r"^-\s*(assert\b|self\.assert)")
ADDED_SKIP = re.compile(r"^\+.*(pytest\.mark\.(skip|xfail)|unittest\.skip)")


def git(*args: str) -> str:
    return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout


def changed_files(base: str) -> list:
    tokens = git("diff", "--name-status", "-z", "--find-renames", f"{base}...HEAD").split("\0")
    rows, i = [], 0
    while i < len(tokens) and tokens[i]:
        status = tokens[i][0]
        if status in ("R", "C"):                 # R100\0old\0new
            rows.append((status, tokens[i + 2]))
            i += 3
        else:                                    # M\0path
            rows.append((status, tokens[i + 1]))
            i += 2
    return rows


def is_test(path: str) -> bool:
    parts = path.split("/")
    name = parts[-1]
    return (bool({"tests", "test"} & set(parts[:-1])) or name == "conftest.py"
            or name.startswith("test_") or name.endswith("_test.py"))


def is_production(path: str) -> bool:
    return not is_test(path) and not path.startswith("docs/") and not path.endswith((".md", ".rst"))


def main(base: str = "origin/main") -> int:
    files = changed_files(base)
    reasons, blocking = [], []
    for _, p in files:
        if (p.startswith(HIGH_RISK_PREFIXES) or p.split("/")[-1] in DEP_MANIFESTS
                or HIGH_RISK_DIRS & set(p.split("/")[:-1])):
            reasons.append(f"high-risk path: {p}")
    edited_tests = [p for s, p in files if is_test(p) and s in ("M", "D", "R")]
    if edited_tests and any(is_production(p) for _, p in files):
        reasons.append(f"existing tests changed alongside source: {edited_tests}")
        diff = git("diff", "-U0", f"{base}...HEAD", "--", *edited_tests).splitlines()
        removed = sum(1 for line in diff if REMOVED_ASSERT.match(line))
        skips = sum(1 for line in diff if ADDED_SKIP.match(line))
        if removed or skips:
            blocking.append(f"{removed} assertion(s) removed, {skips} skip/xfail marker(s) added")
    tier = "high" if reasons or blocking else "low"
    if "GITHUB_OUTPUT" in os.environ:
        with open(os.environ["GITHUB_OUTPUT"], "a") as fh:
            fh.write(f"tier={tier}\n")
    print(f"tier: {tier}")
    for r in reasons + blocking:
        print(f"- {r}")
    if blocking and os.environ.get("TEST_CHANGES_APPROVED") != "1":
        print("Blocked: a code owner must confirm the test changes are intended.")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:2]))

Some deployment details that matter. The three-dot diff needs the merge base, so check out with full history (in GitHub Actions, fetch-depth: 0 on the checkout step). Otherwise git diff fails on a shallow clone. Adapt the prefixes and directory names to your repository’s layout. The design choice is deliberate. High-risk paths are reported but don’t fail the check, because CODEOWNERS and required reviews already enforce human approval for them. Only evidence of test weakening blocks. The override is an explicit, auditable act by a human: your workflow sets TEST_CHANGES_APPROVED=1 only when a code owner applies a designated label. This follows the EvilGenie point. Editing tests is sometimes correct, but it should never happen without someone noticing.

The script has an obvious limitation. An agent can weaken an oracle without deleting an assertion, for example by replacing assert x == expected with assert x is not None on the same line. The diff then shows a removed assert line, so this case is caught. But changing the expected value itself is not. That is why the next layer exists.

6. “Can’t an AI reviewer just absorb the extra review load?”

Partly. The evidence is less encouraging than the marketing.

An MSR 2026 study of code review agents (CRAs; Chowdhury et al., arXiv 2604.03196) found that PRs reviewed only by CRAs had a 45.20% merge rate, against 68.37% for PRs reviewed only by humans, with significantly more abandonment. Among closed CRA-only PRs, 60.2% fell in the 0–30% signal range, and 12 of 13 CRAs had average signal ratios below 60%. Most of what they said wasn’t actionable signal.

Read this carefully. The study is correlational: a lower merge rate doesn’t show that the CRA caused it, since PRs that get only bot review may already be the ones nobody cared about. The signal-to-noise classification is also a keyword-based framework adopted from a blog post, not a validated quality measure.

Vendor evidence points toward depth of review rather than toward removing humans. CodeRabbit’s December 2025 report analyzed 470 open-source PRs (320 AI-co-authored, 150 human-only) and found 10.83 findings per AI PR against 6.45 per human PR. CodeRabbit also reports that high-issue outliers were much more common in AI PRs. The caveats are serious: CodeRabbit sells AI code review, the sample is small, the AI/human labels were inferred, and the “findings” came from CodeRabbit’s own automated tooling. The more robust takeaway is the heavy tail. Most agent PRs may be fine, but some are much worse, which argues for review depth proportional to risk instead of uniform scrutiny.

On the commercial side, GitHub announced at Universe 2026 that Copilot code review Max posted the highest F1 score on ReviewBench for high-severity issues. I couldn’t establish who maintains ReviewBench or whether it is independent, so treat that as a vendor claim until it is reproduced.

One design point worth acting on, labeled as a hypothesis: CRAs often review code produced by the same provider’s models, which raises concerns about closed-loop bias, where a model fails to see the mistakes it tends to make. No study I found shows that cross-provider review catches more defects. It is still a cheap hedge. If your agent runs on one model family, configure your AI reviewer to use a different one, and measure whether its findings differ.

What to do with it: use AI review as a first pass that raises findings for a human to triage, never as the approving reviewer on standard or high tiers. Track its signal ratio yourself by counting how many of its comments lead to a code change.

7. “GitHub’s agent has built-in safeguards. Am I covered?”

Only for GitHub’s own agent, and only partly. The Copilot cloud agent documentation describes a good set of controls:

• The agent can’t mark its PRs “Ready for review” and can’t approve or merge a PR.
• By default, workflows don’t run on the agent’s code until a user with write access clicks Approve and run workflows. This matters because CI is an attack surface: a malicious or confused change to a workflow file can exfiltrate secrets.
• A ruleset rule enabled by default, “Require an additional approval for unattributed Copilot pull requests”, requires one more approval than configured when Copilot opens a pull request that isn’t attributed to a person.
• The user who asked the agent to create the PR can approve it, but that approval doesn’t count toward the required number of approvals. With “Require approval of the most recent reviewable push” enabled, the final push needs n + 1 approvals, where n is the required number.

The catch is that these protections attach to GitHub’s agent and its identity. A local agent such as a CLI tool or IDE agent that commits and pushes under your credentials doesn’t get them automatically. To the platform, its PRs are your PRs. You can approve your own prompt’s output in practice by having a colleague rubber-stamp it, and its CI runs with whatever access your workflows normally get.

The fix is to rebuild the same properties in repository configuration that applies to every author:

1. Separation of duties: rulesets requiring at least one approval from someone other than the last pusher, with “Require approval of the most recent reviewable push” enabled, so a final force-push can’t slip in after approval.
2. CODEOWNERS on high-risk paths, with code-owner review required, including .github/ itself so the gate can’t be edited by the PR it is gating.
3. Required status checks for the gate script, the oracle linter on changed test files, and the mutation job described next.
4. No bypass with personal credentials. Merges go through the ruleset, and agents get scoped tokens instead of a developer’s full permissions wherever your tooling allows.

None of this is agent-specific, and that is the point. Configuration that holds up against an undetected agent also holds up against a tired human at 6 p.m. on a Friday.

8. “What does ‘verification as the job’ look like at serious scale?”

The most concrete industrial evidence comes from Meta, and it relies on mutation testing, not on reviewers reading harder.

ACH (arXiv 2501.12862) generates mutants, deliberately injected faults of a specific kind, and then uses an LLM to write tests that kill them. Applied to 10,795 Android Kotlin classes across seven Meta platforms, it generated 9,095 mutants and 571 privacy-hardening tests. Engineers accepted 73% of the tests and judged 36% to be privacy-relevant.

Just-in-Time Catching Test Generation (arXiv 2601.22832) goes further. It targets risky diffs at review time, generates tests designed to catch bugs in that specific change, and uses LLM-based and rule-based assessors to sort true from false positives. The paper reports that the approach prevented severe bugs from landing and that the assessors reduced human review load by 70%. Engineers mainly confirm whether a behavioral change is intended.

That division of labor is the model to copy. Machines compare behavior against mutations and generated tests. Humans decide intent. The human question changes from “is this code correct?”, which nobody can answer reliably by reading a 40-file diff, to “the tool shows that this change alters behavior X under input Y; did we mean that?”, which a domain expert can answer in a minute.

You don’t need Meta’s infrastructure to get the core benefit. Python mutation-testing tools such as mutmut and Cosmic Ray can run against just the modules a PR touches. Scoping to the diff keeps runtime manageable. A surviving mutant in changed code is a precise, objective review comment: “if I flip this comparison, no test notices.” It also covers the gap the gate script leaves. If an agent changes an expected value to match buggy output, the mutation score on that module usually tells you the tests have stopped pinning down behavior. Configure these tools from their current documentation, and start with the score as an informational check before enforcing a threshold, because baselines vary a lot between codebases.

9. “Open source or commercial: what should the stack look like?”

Compare on technical criteria, not loyalty. The relevant axes are where your code goes (data sovereignty), whether results are deterministic, the noise rate, and the cost of running the tool.

Deterministic verification is where open source is strongest and where the core of the gate belongs:

• Hypothesis for property-based tests, as shown above.
• mutmut or Cosmic Ray for mutation testing on the diff.
• Semgrep Community Edition for custom static rules, which are useful for encoding your team’s “never do this” list, and CodeQL, which is free for public repositories.
• reviewdog, which isn’t an LLM reviewer but aggregates linter output into PR comments.

These tools produce the same answer on every run. A surviving mutant or a falsified property is evidence, not opinion.

LLM review has strong options on both sides. Commercial: GitHub Copilot code review, CodeRabbit, Qodo, Greptile, SonarQube’s AI CodeFix. Open: PR-Agent, originally built by Qodo, moved to a community organization in April 2026. It runs as a GitHub Action without a GPU, calling model APIs through LiteLLM or a local Ollama endpoint. That means you can keep code on infrastructure you control. Kodus is another self-hostable option. Reports about these projects’ licenses conflict across secondary sources, so read the LICENSE file in each repository before you adopt one. Whatever you deploy, confirm with network egress controls, not just configuration, that a “local” reviewer actually stays local.

The technical trade-off is roughly this. Commercial reviewers offer polish, integration and vendor-run benchmarking (independent benchmarks are still missing). Open ones offer auditability, model choice and full control over where code flows, which also makes the cross-provider hypothesis from question 6 easy to test. If sovereignty matters to you, check each commercial vendor’s current self-hosting terms. Whichever you choose, the deterministic layer should be what decides the merge. LLM review should feed human judgment, not replace the oracle.

10. “How do I know whether any of this is working?”

Measure your own pipeline, because platform-wide numbers won’t tell you. A short list of metrics, each computable from data you already have:

• Agent share of PRs in your repositories, using whatever attribution you control: bot identities, commit trailers, branch naming conventions. Expect undercounting. Your local agents won’t label themselves unless you require them to.
• Revert rate and incidents per PR, by origin and by tier. This is the quality signal Octoverse lacks. Git makes reverts easy to find, since git revert writes “This reverts commit” into the message by default.
• Mutation score on changed code, tracked over time and by origin. If agent PRs score consistently lower, your oracle problem is concentrated there.
• Share of agent PRs that modify existing tests, and how often the override label gets applied. A rising override rate means either the gate is miscalibrated or reviewers have started rubber-stamping it.
• Review time per tier. If high-tier reviews are getting faster while agent volume rises, be suspicious, not pleased.
• AI reviewer signal ratio: the fraction of its comments that lead to a code change.

Keep PRs small enough for these numbers to mean something. Given that agent PRs touch multiple files more often, a size limit on agent-authored PRs, plus stacked PRs for larger changes, is one of the cheapest interventions available. A 1,500-line PR with a perfect mutation score is still a PR nobody actually reviewed.

11. “Isn’t ‘verification is the job’ just vendor framing for selling review tools?”

Partly, yes, and it’s worth separating the layers.

Facts: agent PR volume grew sharply on GitHub in 2026 (Octoverse). Passing tests and getting merged overstate quality (METR, 2606.18168, ImpossibleBench). AI-only review correlates with worse outcomes than human review (MSR 2026).

Open debate: whether agent PRs are now as good as human ones. Whether an agent editing tests is cheating or a legitimate fix. Whether cross-provider review reduces blind spots.

Opinion, and this is mine, not a measured result: “verification is the senior engineer’s job” is the right reading of where things are heading. GitHub’s framing supports it, but GitHub benefits commercially from it, so that support counts for little. Meta’s JiT design, where humans judge intent and machines check behavior, supports it more convincingly, because it describes a working production system rather than a slogan.

The career argument follows from the technical one. Producing code is getting cheaper every quarter. What stays scarce is the ability to say precisely what a system must do, encode that as oracles that can’t be gamed, and design pipelines where an unverified change can’t reach production regardless of who or what wrote it. That skill doesn’t depend on any one vendor’s agent winning. It becomes more valuable the more capable the agents get, since better agents produce more plausible wrong answers. An engineer who can tell a team “your tests check almost nothing, here is the mutation report, here is the property that fails” is doing work no PR count captures.

What to do on Monday

1. Run the oracle linter on the test files changed in your last 20 merged PRs. Look at what share of tests have no strong oracle.
2. Add the gate script as a required check, with fetch-depth: 0 and a code-owner-controlled override label.
3. Audit your rulesets against the Copilot cloud agent safeguards: last-push approval, no self-approval in practice, CODEOWNERS on .github/, manual approval of workflow runs from untrusted branches.
4. Pick one critical module and run mutation testing against it. Turn the surviving mutants into the first properties you require.
5. Start logging origin, tier and reverts per PR. In a quarter you will have your own numbers instead of GitHub’s.

Sources

• GitHub Staff, “Octoverse: Commits and pushes reach new highs in 2026 as developers build with agents,” GitHub Blog, Oct 28, 2026. github.blog
• GitHub, “GitHub Universe 2026: Evolving every layer of the stack.” github.blog
• GitHub Docs, “Risks and mitigations for Copilot cloud agent.” docs.github.com
• GitHub Docs, “Reviewing Copilot output.” docs.github.com
• METR, “Many SWE-bench-passing PRs would not be merged into main,” Mar 10, 2026. metr.org
• Empirical study of Claude Code agentic PRs, arXiv 2509.14745. arxiv.org
• Task type and acceptance across five coding agents, arXiv 2602.08915. arxiv.org
• Security-related agentic pull requests, arXiv 2601.00477. arxiv.org
• Test oracle strength in agentic PRs, arXiv 2606.18168. arxiv.org
• Review triage of agentic PRs by type, arXiv 2607.07980. arxiv.org
• Chowdhury et al., “From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents,” MSR 2026 Mining Challenge, arXiv 2604.03196. msrconf.org
• Zhong, Raghunathan, Carlini, “ImpossibleBench: Measuring Reward Hacking in LLM Coding Agents.” lesswrong.com
• EvilGenie, arXiv 2511.21654. arxiv.org
• CodeRabbit, “State of AI vs Human Code Generation,” Dec 17, 2025 (vendor report). coderabbit.ai
• Meta, ACH: mutation-guided LLM test generation, arXiv 2501.12862. arxiv.org
• Meta, Just-in-Time Catching Test Generation, arXiv 2601.22832. arxiv.org
• PR-Agent (community repository). github.com
• Hypothesis documentation. hypothesis.readthedocs.io
• mutmut. github.com · Cosmic Ray. github.com
• Semgrep. semgrep.dev · CodeQL. codeql.github.com · reviewdog. github.com