Run Less, Prove More: Diff-Scoped Test Gating for Agent Patches

An agent patch rarely needs your whole suite. It needs a scope you can defend line by line, and a safety net over everything the scope excludes. Most review pipelines get this backwards: they run everything, watch it go red at random, and then learn nothing about which failure the patch actually caused.

This is a selection strategy, not a speed trick. The gate runs a narrow set of tests, emits a receipt explaining why each test was included, and escalates to the full suite whenever a widening trigger fires. Flaky tests are frozen separately — and a freeze is only granted once a property check covers the same behavior.

The cost is a product, not a number

Write the cost down before optimizing it. For a patch p:

cost(p) = |selected(p)| * t_avg + |frozen| * t_skip  + widen(p) * |full| * t_avg

widen(p) is binary here on purpose. Either a trigger fired or it did not. If you make widening a judgment call, reviewers will quietly skip it under deadline, and your escape rate becomes invisible until production finds it.

Measure t_avg on your own repo. Do not reuse numbers from a blog post — including this one.

Why "only run tests that import the changed file" fails

Naive diff selection is a well-known footgun. Four failure modes matter for agent patches specifically:

1. Shared fixtures. A conftest.py change alters fixtures used by tests that never import the modified module.
2. Runtime wiring. Entry points, plugin registries, DI containers, and string-based imports create edges no static parser sees.
3. Contract drift. An agent changes a serializer or CLI flag; every consumer test still passes locally because none of them are in the import closure.
4. Editable surfaces. Settings modules, migrations, and schema files are imported indirectly by nearly everything.

Each of those becomes a widening trigger in step 3. That is the whole design: be aggressive about narrow selection, be mechanical about widening.

Step 1 — Build a reverse import map

The map is the only expensive part, and it is cacheable per commit.

# scope.py — diff-scoped test selection for a Python repo
import ast, json, pathlib, subprocess

ROOT = pathlib.Path(".").resolve()
SKIP = {".venv", "build", "dist", "node_modules", ".git"}

def changed_files(base: str) -> list[str]:
    out = subprocess.run(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout
    return [f for f in out.splitlines() if f.endswith(".py")]

def module_name(rel: str) -> str:
    parts = list(pathlib.Path(rel).with_suffix("").parts)
    if parts and parts[-1] == "__init__":
        parts.pop()
    return ".".join(parts)

def imports_of(path: pathlib.Path) -> set[str]:
    try:
        tree = ast.parse(path.read_text(encoding="utf-8"))
    except SyntaxError:
        return set()
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            names.update(a.name for a in node.names)
        elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
            names.add(node.module)
    return names

def build_reverse_map() -> dict[str, set[str]]:
    reverse: dict[str, set[str]] = {}
    for path in ROOT.rglob("*.py"):
        if SKIP & set(path.parts):
            continue
        rel = str(path.relative_to(ROOT))
        reverse.setdefault(module_name(rel), set()).add(rel)
        for imp in imports_of(path):
            reverse.setdefault(imp, set()).add(rel)
    return reverse

Store the map keyed by commit SHA. Rebuilding it on every patch is the mistake that makes people abandon the idea.

Step 2 — Compute the importer closure

Walk importer edges upward from the changed modules. Include ancestor packages so pkg.sub also matches pkg.

def importer_closure(seeds: set[str], reverse: dict[str, set[str]]) -> set[str]:
    seen: set[str] = set()
    stack = list(seeds)
    while stack:
        mod = stack.pop()
        for candidate in (mod, *mod.split(".")):
            for f in reverse.get(candidate, ()):
                if f not in seen:
                    seen.add(f)
                    stack.append(module_name(f))
    return seen

def select(base: str = "origin/main") -> dict:
    reverse = build_reverse_map()
    changed = changed_files(base)
    seeds = {module_name(f) for f in changed}
    closure = importer_closure(seeds, reverse)
    tests = sorted(f for f in closure if f.startswith("tests/"))
    return {"base": base, "changed": changed, "selected": tests,
            "widened": None, "reason": []}

A test file is selected because something it imports changed. Nothing else. That is a claim you can audit, which is the point.

Step 3 — Fire widening triggers

Triggers are checked before selection, and any hit short-circuits to the full suite. Keep the list short enough that people actually read it.

Set the fan-out threshold from your own map: print the top 20 modules by importer count and pick a cutoff below the monoliths. On most repos it is embarrassingly obvious.

Step 4 — Cover the excluded surface with properties

Narrow selection is only honest if something else checks the code you did not run. That is what the property layer is for — not coverage theater, but invariants that hold across inputs the suite never generates.

# tests/properties/test_ledger_props.py — illustrative; assert properties your
# module actually promises before adding this file.
from hypothesis import given, settings, strategies as st
from app.ledger import reconcile

rows = st.lists(
    st.fixed_dictionaries({"amount": st.integers(-10**6, 10**6),
                           "ref": st.text(max_size=8)}),
    max_size=50,
)

@settings(max_examples=200, deadline=None)
@given(entries=rows)
def test_reconcile_is_order_insensitive(entries):
    assert reconcile(entries) == reconcile(list(reversed(entries)))

@settings(max_examples=200, deadline=None)
@given(entries=rows)
def test_reconcile_is_total(entries):
    total = sum(e["amount"] for e in entries)
    assert sum(reconcile(entries).values()) == total

These two properties survive refactors that would break every example-based test. They also fail loudly when an agent "optimizes" a fold into a loop that drops the tail.

Step 5 — Freeze flaky tests behind a substitute

A freeze without a substitute is a coverage withdrawal with better branding. Gate it:

freeze(test) is legal only if
  (a) it has an owner and an expiry date, and
  (b) at least one property check covers the behavior the test asserts.

Record both in the receipt. If (b) is unmet, the correct output is a red gate — not a freeze. This is the single rule that keeps quarantine from becoming silent debt.

Step 6 — Emit a scope receipt

Everything above is worthless if the reviewer cannot see it. Write the selection, the triggers checked, the freezes applied, and the property files run into one JSON artifact attached to the patch.

python scope.py --base origin/main --out scope.json
jq -r '.selected[]' scope.json | xargs pytest -q --timeout=300
jq -r '.properties[]' scope.json | xargs pytest -q

A reviewer reading scope.json should be able to answer "what did this gate not run, and why is that acceptable" in under a minute.

Step 7 — Measure the escape rate

Track one number weekly: full-suite failures that the narrow gate did not catch. Run the full suite on a schedule against the same commits and diff the results. If the escape rate is zero for a month, your triggers are probably too conservative — widen less. If it is nonzero, every escape names the missing trigger. Add it.

This is where a free execution lane helps. The narrow gate belongs on the developer's machine because it must be fast; the scheduled full-suite baseline is throughput work that does not need interactivity. MonkeyCode's free model access and free server option fit that second lane: the server runs the scheduled baseline, and the model handles the boring parts — classifying diff hunks into widening triggers and drafting property candidates for human review.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not let a model approve a trigger decision. Classification is a draft; the trigger list is code, and the trigger list is what the gate obeys.

Limitations, and who should skip this

• Static edges only. Dynamic dispatch, getattr chains, and subprocess CLIs are invisible. That is what the trigger table is compensating for, imperfectly.
• Python-shaped. The recipe ports to other languages only if you can build a reliable import graph; dynamic languages and macro-heavy code make it fragile.
• Needs a stable baseline. Without a trustworthy full-suite run to compare against, the escape rate is unmeasurable and the narrow gate is unfalsifiable.
• Skip it for schema migrations, auth and crypto changes, shared integration suites against live external services, and any repo where most tests are end-to-end. Those belong on the full suite every time, and no selection heuristic will save you.
• Skip it if your team will not maintain the trigger list. An unmaintained trigger list is worse than no gate, because it looks like coverage.

The takeaway

Scope is a claim, not an optimization. Write the claim down, make widening mechanical, put properties behind every freeze, and measure how often the claim is wrong. Your agent patches will merge faster for the same reason good tests do: the reviewer stops guessing.

If you try one piece, try the receipt. It is thirty lines of JSON and it changes how the whole gate is argued about.