Marco Rossi·
Fed it our incident timeline and got a blameless postmortem draft with two contributing factors I'd missed while firefighting at 3am
Turns a raw incident timeline into a blameless postmortem with contributing factors, detection gaps, and action items that name systems instead of people.
Blameless Postmortem From a Raw Timeline
You are an SRE writing a blameless postmortem. Convert the raw incident material below into a document the whole company can read without anyone getting thrown under the bus.
Incident timeline (raw notes, Slack fragments, alert timestamps):
{{timeline}}
Impact data (duration, affected users/requests, revenue if known): {{impact}}
Alerts that fired, and when: {{alerts}}
Rules of blamelessness — enforce these strictly:
- Name systems and processes, never people. "The deploy pipeline allowed..." not "X deployed...". If my notes name someone, rewrite around the role or system.
- Every human action in the timeline was reasonable given what that person knew at that moment. If an action looks wrong in hindsight, the finding is about the information they were missing, not the action.
Structure:
1. **Summary** — three sentences max: what broke, for how long, who felt it.
2. **Timeline** — cleaned, timestamped, with DETECTION and MITIGATION moments marked. Compute time-to-detect and time-to-mitigate.
3. **Contributing factors** — not one root cause. List every factor that had to be true for this to happen, including the boring ones (missing alert, stale runbook, config drift). Ask yourself: which factors would ALSO have caused a different incident eventually?
4. **Detection analysis** — what alerted, what should have alerted first, and what a customer would have noticed before we did.
5. **Action items** — each with: type (prevent/detect/mitigate), the system it changes, effort (S/M/L), and a falsifiable done-condition. Reject any action item of the form "be more careful".
End with the three questions the review meeting should debate rather than settle by email.
Ergebnisse
# Postmortem: Checkout API Degradation — June 12
## Summary
Checkout requests failed or timed out for 43 minutes, affecting roughly 18% of attempted purchases. A connection-pool exhaustion in the payments service, triggered by a slow third-party tokenization endpoint, cascaded into full queue saturation. Estimated impact: ~CHF 9,400 in delayed orders, most recovered via retry emails.
## Timeline (UTC)
| Time | Event |
|---|---|
| 02:47 | Third-party tokenization p99 rises from 300ms to 11s |
| 02:52 | Payments service pool reaches max; requests begin queueing |
| 03:04 | **DETECTION** — checkout error-rate alert fires (threshold 5%) |
| 03:09 | On-call acknowledges; initial hypothesis: database |
| 03:21 | Pool saturation identified via connection metrics |
| 03:30 | **MITIGATION** — timeout on tokenization calls reduced 30s → 3s; circuit breaker enabled |
| 03:35 | Error rate below 1%; queues drain |
| 03:47 | All-clear |
**Time to detect: 17 min. Time to mitigate: 26 min from detection.**
## Contributing factors
1. Tokenization calls had a 30s timeout against a vendor whose SLA is 2s — any vendor slowdown converts directly into held connections. *(Would eventually have caused this without any deploy.)*
2. The connection pool and the request queue shared no backpressure signal — saturation was invisible until user-facing errors.
3. No alert existed on pool utilization; the first signal was customer-facing errors, 17 minutes late.
4. The runbook's "checkout errors" page dates from the pre-payments-split architecture, which cost ~12 minutes on the database hypothesis.
5. The vendor's status page showed green throughout — external signal absence extended diagnosis.
## Detection analysis
Customers noticed before we did: first support ticket at 02:58, six minutes before our alert. A pool-utilization alert at 80% would have fired at ~02:50, before any user impact. That is the single highest-leverage change in this document.
## Action items
| # | Action | Type | System | Effort | Done when |
|---|---|---|---|---|---|
| 1 | Alert on payments pool utilization > 80% for 2 min | detect | monitoring | S | Alert fires in staging load test |
| 2 | Reduce tokenization timeout to 3s permanently + circuit breaker with fallback queue | prevent | payments svc | M | Chaos test: vendor stalled 60s, checkout error rate stays < 1% |
| 3 | Rewrite checkout runbook for split architecture | mitigate | docs | S | On-call cold-read drill completes diagnosis < 10 min |
| 4 | Add vendor latency to the checkout SLO dashboard | detect | monitoring | S | Panel live, linked from alert |
## For the review meeting
1. Do we want a second tokenization vendor, or is the circuit breaker + retry queue enough resilience for our size?
2. Should support tickets feed an automated early-warning signal, given they beat our alerting by six minutes?
3. The 5% error-rate threshold met its definition — is the definition wrong?
Modell: Claude Opus 4.6
46 Likes28 SavesScore: 33
2 Kommentare
Ahmed Hassan·
Which factors would ALSO have caused a different incident — that reframing is what separates real postmortems from paperwork.
Daniel Cohen·
Rejecting be-more-careful action items should be a linting rule for postmortems everywhere.