My Tests Are Green. I Broke the Code 10 Ways on Purpose — 5 Broken Versions Still Passed.

There's a trend topic on this site right now called "Verifying AI Agent Patches Beyond CI Green Gates", and the title alone tells you where the conversation has moved. The agent wrote the patch. CI went green. Everyone is asking the same question: so what? Green means the tests passed. It says nothing about whether the tests were worth passing.

I can't run anyone else's agent patches through a laboratory. But I can do the smaller, ruder version of the experiment on my own code: take a function whose test suite is fully green, break the function ten different ways on purpose, and count how many broken versions the suite catches. Mutation testing, done by hand, with a text editor and a grudge.

The victim was a pricing function — the sort of code an agent writes for a shop without breaking a sweat:

def total_with_discount(items, coupon=None):
    total = sum(price for _, price in items)
    if coupon == "SAVE10":
        total = total * 0.9
    if total >= 100:
        total = total - 5
    return round(total, 2)

Plus a bill splitter with a guard clause. Four honest tests. All green: 4 passed in 0.01s.

Then the ten mutations:

Score: 5 killed, 5 survived. Half of my deliberate sabotage walked straight through a green suite.

Read the survivors again, because they're the embarrassing part. Mutant 10 charges customers extra instead of discounting them and no test blinked — my tests never once put a cart over the 100 threshold, so the entire bulk-discount branch could have been replaced with a random number generator and CI would still be green. Mutants 5 and 6 survived because my test values happened to divide evenly; the rounding code was decoration. Mutant 2 and 3 survived for the same reason as 10: a branch nobody exercised.

The killed ones are just as instructive. The suite was excellent at guarding the paths it walked. It caught the typo'd coupon, the broken guard, the lazy sum. A test suite is a fence around exactly the ground your tests have stood on, and a desert everywhere else.

Why this is an AI story now

A year ago, "the tests pass" mostly meant a human wrote code and a human wrote tests that misunderstood the code in compatible ways. Now an agent writes the patch and, increasingly, an agent writes the tests too — sometimes in the same session, from the same reading of the same ticket. If the agent misunderstands the requirement, it will misunderstand it in both files, consistently, and CI will go green with total confidence. My ten mutants are what that looks like from the inside: not sloppy tests, agreeable ones.

The fix isn't more tests in the same style. It's a different kind of check:

• Test the values nobody tried. Every survivor lived in a branch my inputs never reached. One cart at 101 would have killed mutants 2, 3 and 10 in a second.
• Mutate one thing and see. You don't need a mutation framework for the cheap version. Change a constant, flip a sign, delete a round(). If the suite stays green, the suite just told you something true about itself.
• Ask who wrote the tests. A patch and its tests from the same mind, human or machine, share the same blind spot. The green checkmark can't see it. You have to look from outside the pair.

I added the missing branch tests after this, mostly out of shame. The suite is better now. But the number I keep thinking about is the 5 — half of everything I could think to break, in twenty lines of code I wrote myself and understood completely. Imagine the fraction in code nobody understands, shipped because a dashboard turned green.

P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam