Our once-a-week phantom crash got a ranked root-cause list and a stress-test plan

## Heisenbug Hunting Plan — intermittent 502s under load **Bug class:** connection-pool exhaustion (resource leak), not random — correlates with traffic, hides at low load. ## Top 5 root causes (likelihood) | # | Hypothesis | P | |---|-----------|---| | 1 | DB connection not released on the error path in `getUser()` | 0.40 | | 2 | Pool max (10) too low for peak concurrency | 0.25 | | 3 | A slow query holds connections past the 30s idle timeout | 0.15 | | 4 | Keep-alive mismatch between LB and Node | 0.12 | | 5 | Event-loop block from sync crypto stalls release | 0.08 | ## Instrumentation (do first) - Log `pool.totalCount` / `pool.idleCount` every 5s — watch idle trend toward 0. - Add a `queryId` + duration log around every `client.query`, flag any held > 5s. ## Stress strategy Replay production traffic at 3× with `k6`; the leak that takes hours in prod should surface in minutes. ## Defensive fixes - Wrap every checkout in `try/finally { client.release() }` so errors can't leak a connection. - Set a `statement_timeout` so a stuck query frees its slot. ## Elimination Bisect by disabling the suspect endpoint behind a flag — if 502s vanish, hypothesis #1 is confirmed.