Why I'm Building My Own Homelab Operations Center (and What It's Taught Me About Testing So Far)
A project in progress, not a finished product — an internal alpha, actively being hardened before any next step. This article describes how it's being built, not a finished result.
Why a custom tool instead of off-the-shelf monitoring
A homelab with dozens of Docker containers, several physical machines, and a wide mix of services (media stack, home automation, custom apps) quickly runs into one problem: existing monitoring tools either show too many graphs that don't matter, or too little real context — you can see that something's red, but not why, or what to do about it.
MidgardOps is emerging as my own answer to that problem — an "operations center" for the homelab, built up gradually: system, Docker, storage, network. (In the code and in the Docker containers, the project is called heimdall — a working/internal codename left over from an early stage of development; the product itself is called MidgardOps.) It's not a toy on the side — it has real agents deployed on managed hosts, real remote actions (start/stop/restart a service). Important for understanding this article correctly: the app is currently at a stage its own documentation labels "Core Baseline / internal alpha" — actively being hardened, not a finished product deployed and ready for public use. Part of what's described below is therefore more about the process of building the app than about a finished result.
The rule that shaped the app the most: "verify it for real"
The most characteristic pattern across the whole development isn't any one feature, but a recurring ritual: after work that already looked done and tested, comes the question "go through it again, you'll definitely find something" — and almost every time, something turns up.
One particular evening shows this best. After finishing several features that had passed their own test suite and a visual check, an explicit re-audit of that same work followed. Result: three more real problems the previous round had missed:
- The diagnostics copy button silently didn't work on the real deployment — the app runs over plain HTTP (not HTTPS), and the browser's clipboard API requires a secure context. The backend call went through fine (a 200 OK in a real log entry from the user's phone), but the actual copy would fail without any visible error.
- A real performance regression — a feature extended just a few hours earlier was being called from practically every endpoint in the app, not just where it was originally needed, adding half a second to nearly every response. Fixed with a simple short-lived cache.
- This specific deployment's domain name had leaked straight into the app's source code — even though the app is deliberately built as a product for anyone, not tied to one particular machine. This one was caught by the app's own regression test that exists for exactly this purpose.
The lesson, written straight into the project's internal log: "verified live" in practice always turned out to mean something narrower than it looked at first glance. Asking for an explicit re-audit of already-finished work isn't wasted repetition — it's a distinct step that finds things the first round systematically misses.
Bugs that couldn't be found by reading the code
Several of the most serious findings in the app share one trait: they'd have looked perfectly fine in a code review.
A CSRF check that was logically correct and still blocked legitimate requests. The app compares the browser's Origin header against the Host header nginx sends it. The problem: this particular deployment runs on a non-standard port, so the browser includes the port in Origin (on standard ports 80/443 it would leave it out), while nginx was sending the hostname without a port regardless of what port the app was actually listening on — so the two values could never match, no matter how correct the comparison rule itself was. Found only by an actual click in the browser, not by any of the ten original unit tests, none of which ever went through the real nginx configuration.
iOS requires the clipboard copy to happen synchronously. The previous fix for the copy button still awaited a network call to finish before copying — this worked on desktop, but on iOS the browser requires the clipboard API call to happen directly, synchronously, inside the click handler. Fixed by pre-fetching the data ahead of time, so the click itself stays purely synchronous.
Binary framing in Docker logs. When a container runs without an allocated TTY (the common case), the Docker API multiplexes stdout and stderr into its own binary format with an 8-byte header before every line. The app never demultiplexed it — logs showed up with visible non-printable characters at the start of every line. Found visually, straight from a screenshot while testing in a real browser.
A symlink that wasn't resolving where it should have. The system journal size collector ran into the fact that this particular host moves its journal to another disk via an absolute symlink — which, under the bind mount, was being resolved against the container's own root, not the host's. Fixed with a general path-resolution fix, not a hack for one specific case.
The shared conclusion across all four: code can be logically flawless and still not work in the real environment, when it depends on exact values the browser fills in, on OS-specific behavior, or on the infrastructure configuration around the app (nginx, bind mounts). Unit tests over isolated logic systematically miss this — you have to actually run and use the app.
When an audit of the app reveals the app measures the same thing two different ways
One of the bigger architectural findings came from a simple question: why does clicking an incident sometimes show only a generic recommendation instead of a specific diagnosis? The investigation revealed the app had two completely independent systems for deciding "what's wrong" — one for the overall score on the main dashboard, another for the list of specific problems — computing over the same data separately, with their own thresholds. One of them, on top of that, didn't check memory usage at all — a synthetic test with a host at 97% RAM got a clean score from this system without a single note.
Unifying the two systems into one shared logic only happened a few days later, as a separate, deliberately planned task — precisely because a hastily glued-together fix of two parallel status-scoring systems would likely have created more hidden mismatches than it solved.
Security debt that's admitted out loud instead of hidden
An interesting habit that showed up while adding three more data collectors (TLS certificates, backup status, scheduled-job status): each of them needed privileged access to the host system via nsenter. Instead of letting this growing dependency on privileged access go unnoticed, the app has a standalone document with an active warning about exactly this type of architectural debt — and a rule that every new collector of this kind has to be logged there immediately, not after the fact.
This habit — consciously admitting the debt and planning how to eventually move away from it (for example, by shifting to an existing, less privileged mechanism instead of a privileged container) — is exactly what separates an app built with security in mind from an app that just "works."
A big cleanup audit: when an app outgrows its own documentation
After a series of fast increments (root-cause correlation between problems, visibility into failed scheduled jobs, TLS certificate expiry tracking, backup status), a separate, five-round cleanup phase followed:
- Deleted a dead legacy action-execution system — four API routes with zero callers from the frontend, one of which was also leaking environment variables into its response. Cleanly deleted, not kept around "just in case": -866 lines of code.
- Fixed five real bugs found during the audit itself — including a logic error that incorrectly labeled an unknown severity as critical, and duplicate data fetching on one page.
- Removed further dead code — abandoned data-model attempts, empty directories, unused imports. The resulting CSS bundle actually got smaller, confirming the removed code really was dead, not just theoretically unused.
- Added tests for modules that previously had none — a purely additive change, no runtime code was touched.
- Cleaned up documentation so it accurately reflects what's actually done versus still just an idea.
The audit's conclusion matters as much as its results: none of the issues it found called for a structural rewrite — everything was a small, targeted fix. That's a sign the app had been built with reasonable discipline all along, not that it needed rescuing.
What to take away
- "Done and tested" is only ever as good as what was actually tested. An explicit re-audit of already-finished work repeatedly turned up things the first round had missed.
- Logically correct code can be practically broken. The exact headers a browser fills in, how an API behaves on a specific OS, or how symlinks resolve under a bind mount — these can only be verified by actually running the thing, not by reading it.
- Two independent systems measuring the same thing are worse than one imperfect one. As the app grew organically, two parallel ways of computing "what's wrong" drifted apart without anyone intending it.
- Architectural debt is worth logging the moment it's created, not discovering later as a surprise. The rule "every new risky collector has to be logged immediately" is simple, but it works precisely because it's mechanical, not dependent on someone remembering.
- A regular cleanup audit is worth doing even when the app "works." Dead code and drifted systems accumulate quietly — and removing them after the fact is far cheaper than letting them cause a real bug later.
MidgardOps today runs as a live, but still in-progress, operations center over a real homelab — in an internal-alpha stage, with open items (a full owner/"break-glass" access mode, a first-run setup wizard, moving off privileged nsenter access toward a less risky mechanism). It isn't finished because it was written without bugs, but because the process behind it assumes bugs will happen — and builds in a way to find them before they cause anyone a problem.
The same "verify it for real" discipline runs through Sindri (script catalog) and Muninn (document archiving) too.