How Do You Debug a Bug You Can't Trigger on Demand: a Tool for Waiting on Evidence

22 July 2026 · 5 min read

Not a story about a fixed bug — a story about what to do before a bug can even be fixed: build a tool that captures evidence the moment the problem happens to occur.

Two problems that can't be triggered on demand

Using a USB-C dock on a work laptop, two things occasionally happen: not all monitors come up on connecting the dock, and the laptop sometimes can't power off or sleep while the dock is connected. Both problems share a trait that makes them a different kind of animal than an ordinary bug — they're intermittent. Plug the dock in ten times, nine times it's fine, once it isn't. Try to deliberately reproduce it to see it again, and most of the time it just won't happen right then.

The usual debugging approach — run a command, check the output, test a hypothesis — simply doesn't work here. There's nothing to run at the moment the problem happens, because you don't know when it will. So the fix wasn't "let's go fix this right now" — it was "build something that runs quietly in the background and captures evidence exactly when it happens to occur."

How the dock actually connects monitors in the first place

The first step was confirming what mechanism the dock actually uses to connect monitors at all — important for knowing where to even look. The check confirmed native USB-C/Thunderbolt DisplayPort alt-mode passthrough (kernel modules for typec and Thunderbolt), not an extra userspace driver like DisplayLink. That narrowed the suspicion toward the timing of the hardware hotplug process at the kernel level, not some extra software layer on top.

Two separate loops instead of one

The tool runs as a persistent background service (starting on every login) and consists of two independent loops:

Watching raw udev events (USB, DRM, typec, Thunderbolt connect/disconnect) with a precise timestamp.
Periodically checking connected monitors — but only logging when something actually changes, not on every check.

The reason for two separate loops instead of one straightforward one: a single dock connection fires off dozens of udev events at once (every port and interface separately) — tying a log entry directly to each one would instantly flood the log with useless noise. Checking monitor state and diffing it against the previous state handles this naturally — the actual change gets recorded, not every intermediate step on the way to it.

A reference "healthy" run

The tool got a live test during its own creation — connecting the dock while writing the script captured the exact sequence: monitors appeared in the system, initially with no assigned resolution, then filled in to full resolution about five seconds later. The whole process, from the first signal to three fully working monitors, took roughly six seconds.

That captured run now serves as a reference "healthy" pattern. Next time a monitor gets stuck in the in-between state — visible to the system, but with no assigned resolution for longer than a few seconds — the log will show it precisely, instead of it just being a subjective feeling that "it felt slower today."

The second problem needs a different kind of evidence

For the stuck shutdown/sleep problem, a different kind of evidence was needed — not a running log, but a look back at what was happening right before the previous system run ended. Since persistent systemd journaling is enabled, it's possible after a hard reboot to look back at exactly the final moments of the previous run — typically right where something blocked shutdown ("A stop job is running for..."). A separate helper script adds in the tail of the dock-watching log from that same window, so the dock's state at the moment of the hang can be cross-referenced against what the system journal says.

Status: waiting on evidence

Unlike the other articles in this series, this one doesn't end with a solved bug. The tool is deployed and running — waiting for one of the original two problems to happen again by chance, this time with the tool ready to capture exactly what's going on at that moment.

And that's really the whole point of this article: for an intermittent problem, preparing to capture evidence is a legitimate step in its own right — not a delay of the real work. Until you have a way to catch the problem in the act, any fix based on a guess is just guessing. A well-built observation tool sometimes precedes the actual fix by days or weeks — and that's fine.

What to take away

An intermittent problem can't be debugged the same way as a reproducible one. Instead of "run a command and check the output," you need something that runs continuously and waits.
A running log and a retroactive look at the journal are two different tools for two different kinds of problems — one captures a gradual sequence, the other reconstructs the final moments before a failure.
A reference "healthy" run has value on its own — without it, you can't tell when something is genuinely slower or different than it should be, only suspect it.
Debounce/diff logic (log only the change, not every check) is essential when the event source is noisy — otherwise the real signal gets lost in the noise.
An unsolved bug with good observability is a better state than an unsolved bug without it. Preparing to capture evidence is legitimate progress, even before the problem itself is actually solved.