test-audit

Evidence-backed audit of an existing suite, judged on its power to falsify the implementation.

The skill audits a test suite for one thing: falsification power. A suite's value is not that it passes — it is the set of realistic regressions that would make it fail. The audit finds tests that would stay green while the product breaks, proves each claim with quoted evidence, and gives the smallest correction. It is adversarial by design, but credits genuinely strong tests with the same rigor.

Invoke it when someone asks to review, audit, or grade tests ("are these tests any good?", "would these catch a regression?"), wants agent-written tests double-checked, or after any sizable test-writing effort. It is also the mandated self-check step at the end of the test-writing skill.

Scoping

Before judging anything, the audit establishes the suite's goals — the acceptance criteria or stated purpose the tests claim to guard — and pulls in everything load-bearing: the tests, helpers/fixtures/factories, runner and CI registration (an unregistered test is a blind spot with a filename), contract tests, README claims about coverage, and retained run artifacts. It confirms each test actually runs in CI and can actually fail — not skipped, quarantined, or auto-retried into green.

The per-test interrogation

Every test (or tight group) is answered with evidence — quoted code, not paraphrase — against ten questions: fidelity to the spec vs. to what the implementation happens to do; entrypoint honesty (supported product entrypoint, or a bypass?); real collaborators (what is silently faked, and what bug class does each fake delete?); assertion strength (can it detect wrong data, or only presence and truthiness?); a concrete false-pass analysis (the specific breakage that leaves the test green); a false-fail analysis (could it go red for environment reasons before reaching its oracle?); fail-closed integrity (weakened assertions, unreached soft assertions); scenario debt (persistence, RBAC, idempotency, concurrency where the journey requires them); level honesty (contract tests counted as journey proof; claims tested at a level that can't observe them); and a regression ledger — what it would catch, and what nearby it would miss.

Suite-level analysis

Zoomed out, the audit maps scenarios × levels for duplication and blind spots, flags misplaced coverage (cases burning E2E minutes that a lower level fully falsifies), checks determinism and isolation (shared fixtures, order dependence, sleeps, unseeded randomness — anything that trains humans to rerun instead of read), and verifies that README/docs claims about coverage match reality.

Deliverable and severity scale

The output structure is fixed: an executive verdict per stated goal (no hedging), findings ordered by severity — each with a verdict (Misleading, Needs revision, Adequate, or Strong), affected tests with file and line, quoted evidence, the concrete false-pass or flake story, and the smallest correction — then a coverage map and a prioritized remediation plan (must-fix misleading tests first, then real gaps, then confirmed product bugs to keep fail-closed).

The severity calibration is the skill's sharpest design decision:

1. A misleading test — one that would pass through the very breakage it exists to catch — outranks a missing test, because it makes the team more confident than having no test at all. The Misleading verdict is reserved for exactly that, and the audit says so plainly.
2. A missing critical journey outranks a flaky-but-honest test.
3. Style and duplication come last.

Relation to test-writing

The two skills are one loop: test-writing designs tests so the answer to "could this pass while the behavior is broken?" is no, and test-audit is the adversary that checks whether an existing suite actually achieves that — including as the automated self-check before test-writing's teach-back. Run it on inherited suites before trusting their green.