What one year of incident reviews taught us

We wrote twenty-six incident reviews last year. In January somebody suggested reading all of them in one sitting, which sounded like a bad afternoon and turned out to be the most useful thing we did all quarter.

We expected to find a cluster of bad code in one subsystem. That is not what the reviews said.

Detection was slower than the fix

In nineteen of twenty-six incidents, the time from the change landing to somebody noticing was longer, often much longer, than the time from noticing to resolving it. The median was 47 minutes to detect and 11 minutes to fix.

We had spent the year making deploys faster and rollbacks safer. Both were already the quick part. The slow part was nobody knowing yet.

We were optimising the ten minutes we could see and ignoring the forty-seven we could not.

The alerts we had were not the alerts we needed

We had plenty of alerts. They fired on causes we had already thought of: CPU, memory, disk, error rate on a handful of endpoints. Almost none fired on the thing a user would actually notice.

• CPU was normal during eleven of the incidents

• Error rate was normal during fourteen, because the failures returned 200 with wrong content

• The single most predictive signal was a drop in a business metric, which we were not alerting on at all

So we changed what we alert on:

# Before: alert when the machine is unhealthy
- alert: HighCPU
  expr: avg(rate(cpu_seconds_total[5m])) > 0.85
  for: 10m

# After: alert when the product is unhealthy
- alert: PublishRateBelowBaseline
  expr: |
    sum(rate(posts_published_total[15m]))
      < 0.4 * sum(rate(posts_published_total[15m] offset 1w))
  for: 10m
  annotations:
    summary: "Publishing is running well below the same time last week"

Rehearsed rollbacks were fast; unrehearsed ones were not

Where a team had practised the rollback, mitigation took minutes. Where they had not, the review almost always contained a sentence like "we were not sure whether rolling back would lose data". Uncertainty, not tooling, was the delay.

1. Write the rollback plan in the pull request, before merging.

2. Practise it in staging on anything touching data.

3. If the rollback is not safe, that is a design problem to solve now, not at 3am.

The riskiest deploys were the smallest ones

This surprised everybody. Large, obviously risky changes got scrutiny: reviewers, a plan, someone watching the graphs. One-line changes got none of that, and produced seven of our twenty-six incidents.

Nobody schedules a war room for a config tweak. The config tweak does not know that.

What changed

The last one is the cheapest and, we think, the most valuable. Individual reviews tell you about individual failures. Reading twenty-six at once tells you about your organisation, and that is the thing you can actually fix.