Luca Brunner·
Ran a test-gap analysis on our payments module before the audit. Found we had 94% line coverage and still zero tests for the three failure modes that actually matter.
Analyzes a module's code and existing tests to find untested behaviors and failure modes, beyond line coverage.
Behavioral Test Gap Analysis
You are a test engineer who believes line coverage is a vanity metric. I will give you a module's source code and its existing test file(s). Find what is genuinely untested — behaviors and failure modes, not lines.
Module source:
{{module_source}}
Existing tests:
{{test_files}}
Context: what this module does in production and what a failure costs: {{production_context}}
Analysis:
1. **Behavior inventory.** List every externally observable behavior of this module: return values per input class, side effects (writes, emits, calls out), thrown/returned errors, and state transitions. Number them B1, B2, ...
2. **Test mapping.** For each behavior, name the existing test(s) that cover it, or mark UNTESTED. Be strict: a test that merely executes the code path without asserting on the behavior's outcome does not count — call those out separately as "executed but unasserted".
3. **Failure-mode gaps.** Specifically check for missing tests around: error paths (what happens when the dependency throws / times out / returns malformed data), boundary values (empty, zero, negative, max, unicode, duplicate), concurrency or idempotency assumptions (what if this runs twice?), and partial-failure states (side effect A succeeded, side effect B failed — what is the system state?). For each gap, describe the production scenario it corresponds to.
4. **Ranked test plan.** Order the missing tests by (production cost of the failure × plausibility). Top 5 get full test skeletons in {{test_framework}} with descriptive names, arrange/act/assert structure, and the specific assertion that matters. The rest get one-line descriptions.
5. **Deletions.** Any existing tests that assert nothing meaningful or duplicate another test — recommend removal. Test suites also accrue debt.
Ergebnisse
## Test Gap Analysis: `payments/capture.ts`
### Behavior Inventory (12 found, key ones)
- B1: successful capture → returns receipt, writes `captured_at`, emits `payment.captured`
- B4: provider timeout → retries once, then returns `CaptureRetryable`
- B5: provider returns `insufficient_funds` → returns `CaptureDeclined`, does NOT retry
- B7: capture called twice for same intent → second call returns original receipt (idempotency)
- B9: DB write fails AFTER provider capture succeeded → ??? (code swallows the error, logs, returns success)
- B11: amount = 0 → currently proceeds to provider (!)
### Test Mapping
| Behavior | Coverage |
|---|---|
| B1 | ✅ `captures a valid payment` |
| B4 | ⚠️ executed but unasserted — `handles timeout` only asserts no throw, never checks the retry happened or the return type |
| B5 | **UNTESTED** |
| B7 | **UNTESTED** — the single most load-bearing behavior in a payments module |
| B9 | **UNTESTED** — and the current behavior is arguably a bug |
| B11 | **UNTESTED** |
Line coverage is 94% because the happy path executes almost every line. Behavior coverage: 7 of 12.
### Failure-Mode Gaps (production scenarios)
- **B7 gap:** webhook redelivery WILL call capture twice — without an idempotency test, any refactor can silently introduce double-charging.
- **B9 gap:** provider captured, our DB didn't record it → customer charged, order unpaid in our system. This is the exact state behind your two worst support tickets last quarter.
- **B4 gap:** unasserted retry means the retry could be deleted and tests stay green.
### Ranked Test Plan (top 3 skeletons, vitest)
```ts
describe("capture idempotency", () => {
it("returns the original receipt when called twice with the same intent id", async () => {
const first = await capture(intent);
const second = await capture(intent);
expect(second.receiptId).toBe(first.receiptId);
expect(providerMock.capture).toHaveBeenCalledTimes(1); // the assertion that matters
});
});
it("surfaces a reconciliation state when DB write fails after provider success", async () => {
dbMock.update.mockRejectedValueOnce(new Error("conn reset"));
const result = await capture(intent);
expect(result.status).toBe("needs_reconciliation"); // current code returns 'success' — fix first
});
it("does not retry on insufficient_funds", async () => {
providerMock.capture.mockResolvedValueOnce(declined());
await capture(intent);
expect(providerMock.capture).toHaveBeenCalledTimes(1);
});
```
### Deletions
- `test: capture works` — duplicates B1 test with weaker assertions. Remove.
- `test: logs on error` — asserts a logger call, couples tests to log format. Remove or fold into B9 test.
Modell: GitHub Copilot
34 Likes18 SavesScore: 24
2 Kommentare
Emily Chen·
'Executed but unasserted' deserves its own coverage metric. The timeout test asserting nothing is every codebase I have ever joined.
Daniel Cohen·
B9 is a real vulnerability class too — state desync between provider and system of record. Good catch pattern.
