Reading logs like a detective

Logs are a crime scene, not a novel

You do not read them front to back. You look for the moment the story changed. The skill is knowing where to stand so the important line finds you.

Narrow before you widen

In Splunk I start narrow — a single trace or request id — and widen only when I need context. Searching everything first buries the signal. A tight time window plus one strong filter beats a broad query every time.

• Start from a known-bad request, then expand outward.
• Filter by status, host, or endpoint before by keyword.
• Let the dashboard show you rate of change, not just totals.

Bring the host into it

Application logs tell you what the code saw; the host tells you what the machine felt. A quick pass with grep, journalctl, and a look at resource pressure often explains an error the app logs only hint at. The two views together are far stronger than either alone.

Automate the boring 80%

Any investigation step I do twice becomes a small Bash script — collect the last N minutes of logs, pull the health check, grep for the usual suspects. The script does not solve the incident, but it means every investigation starts from evidence instead of from scratch.

Write down what the logs could not tell you

The gaps are as useful as the hits. If I could not find the answer because a service logged nothing useful, that is a finding — and usually the first line of the next runbook.