Compare a good run and a bad run of your logs

When something worked yesterday and fails today, the answer is usually in the lines that only appear in the bad run. This guide shows how to find them with Log Compare, using a small example you can follow.

When this helps

• A release broke something, and you have logs from before and after it.
• A batch job or a test that passed last week fails now.
• A request works for one customer and fails for another.

Five steps

1. Export the good run as plain text from your log tool. Use a run that worked, such as the previous release or yesterday's batch.
2. Export the bad run from the same application, ideally the same kind of request or job.
3. Drop both files into Log Compare. The comparison runs as soon as both are loaded.
4. Read the New tab first. These are messages only the bad run has. Then Missing: messages the good run has that the bad run lacks, which often means a step never finished. Then Changed.
5. Open a row to see one example from each run, and check the "When it happened" strip. If the errors cluster near the end, the failure started there.

A worked example

Take an order service. Both runs handle 240 orders, one after another. In the bad run, every fourth order hits a slow payment provider. Log Compare reports these rows, in the order it ranks them:

Tab	Message (numbers replaced by #)	Good run	Bad run
New	WARN payment provider slow, retry # for order #	0	60
New	ERROR payment provider timeout after # ms for order #, with its two stack-trace lines folded in	0	30
Changed	INFO order # shipped	240	180
Changed	INFO order # paid in # ms	240	210

Reading the rows together tells the story:

• The 60 new warnings are the slow payments. Each one is followed by a retry.
• Half of those (30) timed out, and only those show a new error with a stack trace. The timeouts all happen in the second half of the bad run, which the "When it happened" strip shows.
• The 60 orders missing from "shipped" are the same 60 slow-payment orders. They never reached shipping in the bad run.
• The other 30 slow payments were paid after a long wait (900 ms or more), so "paid in" is still there but with longer times.

Now you know where to look: the payment provider call, its timeout and the retry, not the shipping or the cache.

Tips for better results

• Compare logs from the same application and the same kind of request. Mixed traffic makes the differences harder to read.
• Numbers, ids and quoted text are ignored, so order 4411 and order 9020 count as one message.
• Stack-trace lines are kept with their error, so one failure is listed once, not once per line.
• Add a third run, for example a second bad run. A message that appears in both bad runs and not in the good one is much more likely to be the real cause.

Your files are read in your browser and never uploaded.

Questions

What is a good run and a bad run?

Two logs of the same application: one where things worked, such as yesterday's batch or the release before, and one where they failed. Comparing them shows the messages that only the bad run has, which is usually where the problem starts.

Which runs should I compare?

Pick two runs that are as alike as possible: the same kind of request, job or test, from the same service. Differences in what was asked make more noise than differences in how it ran.

How big can the logs be?

Each file can be large, up to a few hundred MB, and is read in your browser. Very large logs take longer to read, so you can narrow them to the time window that failed before you export.

Can I use logs from Kubernetes or Graylog?

Yes, as plain text. Export the lines from your log tool first, for example one file per pod or per service. Structured JSON exports are not read by this version.

Are my logs uploaded?

No. Your browser reads both files on your device. Nothing is sent or saved, and closing the tab clears them.