Triaging a production incident: alert to resolution

The goal is evidence, not heroics

When an alert fires at 3 a.m., the worst thing you can do is start guessing. The job is to move from "something is wrong" to "here is exactly what is wrong, and here is the proof" as quickly as possible. Everything below is in service of that.

Confirm the blast radius first

Before touching anything, I answer three questions: is this one customer or many, is it degraded or fully down, and when did it start. The monitoring dashboard usually answers all three in under a minute — a spike in error rate lined up against a deploy marker tells a very clear story.

Read the logs like a timeline

Logs are most useful when you stop reading them as text and start reading them as a sequence. I anchor on the first bad event, not the loudest one. The error a user reports is often three hops downstream of the real cause.

• Find the earliest anomaly, not the noisiest.
• Correlate the log timestamp against the metric that moved.
• Note what is normal so the abnormal stands out.

Reproduce, then escalate with a package

If it needs engineering, they should never have to redo my investigation. A good escalation is a small package: the reproduction steps, the relevant log lines, the timeline, and the impact. Half the resolution time is often just handing over clean evidence.

Close the loop

A fix is not done until the affected client hears it from a human. Verify the fix in the same dashboard that caught the problem, then tell the customer in language that matches who they are — technical for the technical, plain for everyone else.