What is False green?
A test or check that passes even though the behaviour it claims to guard is broken — for example, one that stays green when the fix is reverted.
Automated work multiplies checks; a check that cannot fail gives confidence without protection.
On this page
A false green is a passing check that fails to constrain what it appears to check: the test asserts too little, asserts the wrong thing, or mirrors the implementation's own misunderstanding — so the suite stays green while the behaviour it was meant to guard is broken. The name comes from the diagnostic that exposes it: revert the fix, and the test stays green. A check that cannot notice the break was never checking it.
False greens predate AI coding, but agent-driven work multiplies them, because the author of the change and the author of its checks are so often the same reader of the requirement.
Why agents produce more of them
Three mechanisms, and all three are amplified when an agent writes the code.
The author-writes-its-own-tests loop. When the same author — human or model — produces the change and its tests, both descend from one reading of the requirement. If that reading is wrong, the code and the tests are wrong in the same direction, consistently, and nothing in the run objects.
The satisficing pull. An agent optimises toward signals it can observe. A weak assertion turns green sooner than a complete one, and iteration pressure rewards sooner. Nothing in "make the test pass" says "make the test able to fail".
The oracle problem. A test compares behaviour against some definition of correct. When that definition lives only in the implementer's head — or in the implementer's own generated test — the comparison proves nothing about the requirement.
The research on AI coding benchmarks put numbers on this. Wang, Pradel and Liu (2025) re-examined patches counted as correct on SWE-bench Verified and found 7.8% actually fail the developer-written test suite, while 29.6% behave differently from the reference fix — inflating reported resolution rates by 6.2 percentage points. UTBoost (ACL 2025) strengthened inadequate test suites with generated user tests and flipped the verdict on 24.4% of affected instances — entire curated suites that could not tell a correct patch from a broken one.
What weak versus strong checks look like
| Weak | Strong |
|---|---|
expect(response.status).to eq(200) on an endpoint that should refuse unauthenticated callers |
The missing-key and invalid-key cases each assert the exact failure status and error body |
| Asserting a field exists in a payload | Asserting the field's value for a known input, plus the absent case |
| Tests generated from the diff after the code was written | Tests derived from the acceptance criteria before implementation details existed |
| A suite that has never failed since it was written | A suite someone has watched fail for a realistic break |
The one-question version of the mutation-testing criterion is enough for everyday use: if this code were broken in the way the requirement fears, would this test notice? Google's experience with mutation testing at industrial scale — deliberately breaking code to see whether tests notice, with results surfaced as review comments across 24,000+ developers and 1,000+ projects — shows the discipline scales when it rides the review loop rather than demanding separate reports.
Where false greens hide in agent workflows
Beyond the test suite, every automated verdict in the pipeline can carry a false green:
- Self-reported completion. "Done" claimed by the authoring run, unchecked, is the purest form: a green with no checker behind it.
- Review that echoes. A reviewing agent that shares the author's context, or worse its identity, rubber-stamps rather than checks.
- Verification without evidence. "Tests pass" as a sentence, with no counts, no output, no record — unauditable, therefore unfalsifiable.
The common defence is structural, not vigilant: independent review, tests tied to acceptance criteria written before the code, and evidence attached to every verdict — the record that lets an auditor distinguish a check that ran from a check that was claimed.
False greens in YATFA
YATFA assumes the suite is the weakest link and builds past it: review is performed by an agent that did not write the change, QA verifies against the ticket's acceptance criteria on a live system, and the evidence — requests, output, counts — is attached to the ticket itself. When an article in the knowledge base records a lesson about a false green, the next run starts from it; see living knowledge and the full write-up in testing AI-generated code.
Frequently asked questions
How do I know if my tests contain false greens?
The cheapest probe is the revert test: for a sample of tests, revert the behaviour they claim to guard and watch whether they fail. Tests that stay green are decorative. Mutation tooling automates this at scale; the revert test is the manual version worth running on any suite you depend on.
Are AI-written tests worse than human-written ones?
Not inherently — but the failure mode is more likely, because the author-writes-its-own-tests loop is the default. Human teams have the same risk and mitigate it with review culture; agent setups need the equivalent structure: tests tied to the brief's criteria and a reviewing agent that did not share the author's reading.
Does more test coverage fix this?
No. Coverage measures which lines executed, not what the assertions proved. A suite can cover every line and still be a field of false greens if each test asserts merely that nothing crashed. Assertion strength — would it notice a realistic break — is the axis that matters.