How to read production logs across several services

When a system fails across several services, no single log usually tells the whole story. A web server reports a 504, the application reports a timeout, and the database reports nothing unusual. Reading those logs well is mostly a matter of putting them on one timeline, following one request through them, and noticing what changed. This guide explains how, and it works for any stack that writes timestamped text logs.

What a log line usually contains

Most logs follow the same shape: a timestamp, a severity level, a source (a service, pod or host), and a message. Some add a request id, trace id or user id. Level names such as DEBUG, INFO, WARN and ERROR tell you how serious an event was, but teams use them differently, so a WARN in one service can matter more than an ERROR in another. Read the messages, not only the levels.

Step by step

1. Collect the logs for the same time window. Pull each service's log for the same period, for example ten minutes before the first error to ten minutes after the last. For Kubernetes pods, kubectl logs --timestamps adds a timestamp to every line, which makes merging possible later. Log platforms such as Graylog, Loki or Elasticsearch can export a search as CSV or plain text.
2. Find the errors that repeat. A busy service can write thousands of similar error lines. Lines that differ only in an id, a number, an IP address or a time are the same problem. Group them, and the most frequent group is usually the first thing to investigate. Be careful: the most frequent error is not always the root cause. It can be a symptom that happens often.
3. Put the logs on one timeline. Sorting each file by time and reading them next to each other shows the order of events. Check that the clocks agree. A server whose clock runs two seconds fast can make a cause look as if it happened after its effect. Compare the clocks with a known event, such as a deployment, or check whether the servers use NTP.
4. Follow one request through every service. Find a request id, trace id or order number in one error, and search every log for it. A single failed request usually leaves a trail: the request arrives, calls the payments service, waits, and finally returns an error. The point where the trail stops, or where the waiting starts, is where to look next.
5. Compare with a run that worked. If you have logs from before a change or from a healthy period, compare them with the failing period. Look for messages that are new, messages that have disappeared, and messages whose counts changed sharply. A new warning that appears only in the bad run is a stronger clue than an error that was always there.
6. Check the order of events, not one line. A timeout in the payments service at 10:14:02.9 followed by a 504 at the web server at 10:14:03 suggests the web server was waiting on payments. A lock wait in the database log just before the timeout would point further down the chain. The sequence is the evidence; a single line rarely is.

Things to watch for

Time zones. One service may log in UTC and another in local time. Mixed zones look like a clock problem, and they are easy to misread. Note the zone of each file before you merge them.

Multi-line messages. A stack trace is many lines but one event. Read it as a block, or the exception that caused the error can be split from the message that describes it.

Retries and reconnects. A burst of "connection reset" or "retrying" lines often shows the first failure, not the cause of the spike. Look at what happened just before the burst.

Sensitive data. Logs often contain customer emails, card numbers, tokens or session ids. Remove or mask them before you share a log in a ticket, a chat or a screenshot.

If you want to try it on your own logs

You can do every step above with a text editor and command-line tools such as grep, sort and awk. If you would rather not write those commands, our free browser tools do the same work on your own device, and nothing is uploaded: the Log Playground opens several logs side by side, merges them by time and groups repeated errors, and the Log Compare tool lists what is new or missing between two runs. For CSV exports, the Log Sorter handles the sorting and filtering. The good run and bad run guide and the guide to merging pod logs go deeper on two of these steps.

Questions