Trigger: outage, security event, data breach, P1/P2 defect (classes in POL-08). Owner: Jonny Allum. Policy anchors: POL-08, POL-02 §5.
0. First five minutes
- Classify (P1/P2/P3 — POL-08 §1). Security suspicion ⇒ treat as P1.
- Start the incident note immediately (timestamped scratch file — times, observations, actions). You will not remember the order later.
- Is customer data actively at risk? If yes → containment before diagnosis.
1. Containment moves (pick what applies)
| Situation | Move |
|---|---|
| Credential/secret suspected leaked | Revoke/rotate it NOW (SOP-09 §4); investigate after |
| Unknown session/key on an account | Revoke first, investigate second (POL-04 §3) |
| App misbehaving dangerously | pm2 stop the app — an honest outage beats corrupt writes |
| One tenant affected by a data defect | Disable the module for that tenant via Section Builder rather than platform-wide stop |
| Runaway automation/cron | Disable the cron line (comment the schedule), then fix |
| Payment anomalies | Pause the webhook endpoint in Stripe; reconcile before re-enabling (SOP-08) |
Preserve evidence where feasible: copy Caddy/PM2 logs, note Supabase and Stripe event ids before restarting things (POL-08 §3).
2. Diagnose and fix
- Reproduce locally against file-backed stores where possible.
- Fix forward with the smallest diff; gates still run (POL-06 emergency class); deploy per SOP-03 with full smoke test.
3. Communications
- P1: affected tenants get a plain-language status within 4 hours — what's broken, what it means for them, next update time. Use the critical alert email path or direct email.
- Data breach: POL-02 §5 clocks — tenants without undue delay; ICO ≤72h where required (controller side); individuals where risk is high. Draft honestly: what data, whose, what window, what's done, what they should do.
- P2/P3: tell affected tenants when fixed, or sooner if they noticed.
4. Close-out (within 3 working days of resolution)
- ☐Incident record completed: timeline, root cause, fix commit hash, comms sent
- ☐Retrospective: why didn't the gates catch it? What control/SOP changes?
- ☐Follow-up actions filed and dated (not "someday")
- ☐Risk register updated if a new risk surfaced (POL-11)
- ☐Breach log updated (notifiable or not) if personal data was involved