Marco Rossi·
Dumped four weeks of PagerDuty exports into this. It clustered our alerts, found six rules causing 70% of pages, and drafted new thresholds. Quietest on-call week since March.
Analyzes an alert/paging export to find the noisiest rules, classify them by failure mode, and propose threshold or routing changes with explicit safety arguments.
Alert Noise Autopsy
You are an SRE doing an alert noise autopsy. Input: a paging export and our SLO definitions. Output: which alert rules to change and a defensible argument for each change — because every threshold loosened is a bet that needs a safety case, not just a desire for sleep.
Alert export (CSV or JSON — rule name, timestamp, duration, ack time, resolution, whether it paged):
{{alert_export}}
SLO definitions and their error budgets: {{slo_definitions}}
Analysis steps:
1. **Concentration.** Rank rules by pages generated. Report the cumulative share — how many rules cause 50% and 80% of all pages. This number decides whether we tune rules or rebuild the philosophy.
2. **Classify each top-10 rule** by failure mode: (a) flappy — fires and auto-resolves quickly, (b) redundant — consistently fires within minutes of another rule for the same underlying cause, (c) non-actionable — acked but no action taken, resolves on its own, (d) mis-routed — actioned, but by someone other than the team paged, (e) genuine — paged, actioned, mattered. Use ack-to-resolution patterns and durations as evidence; state your inference logic.
3. **SLO alignment.** For each noisy rule: does it protect an SLO? A rule that pages but maps to no SLO and no user-visible symptom is a candidate for demotion to ticket. A symptom-based SLO burn-rate alert beats three cause-based alerts — identify where that swap applies.
4. **Proposals.** For each change: current behavior, proposed behavior (new threshold with hysteresis, demotion to ticket, deletion, or merge into a burn-rate alert), and the safety argument — what real incident from the export would this change have delayed or missed, and by how much? If the answer is 'one, by 20 minutes', say so honestly.
5. **Verification plan.** How we validate in 30 days: expected page volume, and the tripwire that reverts the change.
Ergebnisse
## Alert autopsy: 4 weeks, 312 pages, 47 distinct rules
**1. Concentration.** 6 rules generate 218 pages (70%). 3 rules generate 51%. This is a tuning problem, not a philosophy problem — good news, the fix is surgical.
**2. Top offender classification:**
| Rule | Pages | Class | Evidence |
|---|---|---|---|
| `disk-usage-warn-85` | 61 | (c) non-actionable | median ack 2 min, action taken in 3/61 cases, all self-resolved via log rotation |
| `api-p99-latency-spike` | 49 | (a) flappy | 41/49 auto-resolved < 5 min; fires on single-window spikes during deploys |
| `pod-restart-count` | 38 | (b) redundant | fires within 4 min of `api-5xx-rate` in 33/38 cases — same underlying cause |
| `db-connections-high` | 31 | (e) genuine | 28/31 actioned with real intervention; keep, do not touch |
| `queue-depth-worker` | 24 | (d) mis-routed | pages platform team; resolved by app team in 21/24 cases |
| `cert-expiry-30d` | 15 | (c) non-actionable as a PAGE | zero urgency at 30 days; every ack deferred |
**3. SLO alignment.** `api-p99-latency-spike` and `api-5xx-rate` both protect the availability SLO — replace both plus `pod-restart-count` with two burn-rate alerts (fast: 14.4x over 1h, slow: 3x over 6h). `disk-usage-warn-85` maps to no SLO and no symptom: demote. `cert-expiry-30d`: ticket at 30d, page only at 7d.
**4. Proposals with safety arguments:**
1. **Demote `disk-usage-warn-85` to ticket; page at 95% with 10-min sustain.** Safety: zero incidents in export originated from disk; the one genuine disk event (June 12) went 85→97% in 4 hours — the 95% page still gives ~90 min. Honest cost: a pathological fill (<30 min) would page ~25 min later than today. None observed in 4 weeks.
2. **Burn-rate consolidation** (3 rules → 2). Safety: replayed against the export, every genuine availability incident (5) still pages within 4 min of current behavior; deploy-window flaps (41 pages) disappear entirely.
3. **Re-route `queue-depth-worker` to app team** + add runbook link. No detection change, pure routing.
4. **`cert-expiry` split** as above. Safety: renewal takes 15 min; 7 days is 670x margin.
**Projected volume: 312 → ~95 pages/4 weeks (-70%).**
**5. Verification.** Tripwires for 30-day review: any SLO-impacting incident detected late by >5 min vs. old rules → revert consolidation; disk past 95% without prior ticket action → revert demotion. Calendar reminder, not vibes.
Modell: Mistral Large 2
25 Likes16 SavesScore: 19
2 Kommentare
Ahmed Hassan·
Replaying proposals against the historical export to build the safety case is the credibility move. 'Would have missed one incident by 20 minutes, honestly stated' beats any confident assertion.
Tobias Keller·
The flappy-during-deploys pattern is universal. Single-window latency alerts with no sustain period are just deploy notifications with extra panic.