Every change I make to my own systems ships with a verification step: a command, and the output it should print. A good share of the defects I’ve found this year weren’t in the changes. They were in those checks. Each one ran, printed something that looked like success, and let the thing it existed to catch go straight past.
These are six of them, and the handful of habits they left behind.
A count that read as a match
One check compared the first N lines of two files by hashing both. It worked where I wrote it, on Linux. On macOS, wc -l pads its output with spaces, so N came out as " 417", head rejected it as an invalid line count, and both sides of the comparison hashed empty input. Two empty inputs hash the same, so the check printed MATCH, and it did so precisely on the branch the change existed to fix.
The check even had a negative control: two files that should hash differently. But the control passed a literal number where the real check computed N, so it proved the hashing worked and never touched the part that was broken.
The habit: route the negative case through the same variables as the positive one, and assert the derived value as well as the result. A check that also says “N must be 417” fails loudly instead of matching on nothing. It helps to know the empty-input hash on sight; for SHA-256 it starts e3b0c442.
“Non-zero” when the answer was seven
A check for a new sign-in configuration counted the matching variables inside a container and expected “non-zero”. It returned 5. The configuration needed 7, and the two missing were the two secrets that mattered.
Two things were wrong. Running a command inside a container starts a fresh process from the image’s environment, so it never sees variables the container’s own start-up script exported. And a threshold of “more than zero” can’t tell a half-configured service from a working one.
The habit: assert the exact number, or the exact names. Then read the running process’s own environment, from the place that can actually see it.
A permission error that printed 0
The fix for the check above had a trap of its own. Reading a root-owned process’s environment with sudo tr '\0' '\n' < /proc/1/environ looks right, but the redirect is done by the unprivileged shell before sudo runs. The file can’t be opened, tr never starts, the error goes to stderr, and the count on stdout is a perfectly plausible 0, on a host where everything is correct.
The habit: when a check needs privilege, put the whole command, redirect included, inside the privileged shell. A zero is only evidence if the command that produced it could have produced something else.
A deletion gate that could never pass
A check meant to prove a change only added lines counted diff lines starting with -, expecting zero. Every diff has a --- a/file header, so it could never read zero. The obvious repair, skipping lines that start with two dashes, is worse: a deleted Markdown bullet shows in a diff as -- item, so the repaired check can’t see the deletions it exists to catch.
The habit: ask the tool for the number directly (git diff --numstat gives added and deleted counts per file), and give the check a control that really deletes a line.
A search that looked at the wrong branch
Three checks in one day reported that work was missing. It wasn’t: every change was on the main branch. The clone they searched had been left on an older feature branch, by an instruction in the same plan, and a search reads whatever is checked out.
This is the worst direction for a check to fail in, because “the work was never done” is believable. The habit: name the branch each check assumes, and where the checkout might be elsewhere, read the file from that branch explicitly instead of from the working tree.
An expected count nobody had counted
Three plans in a row told whoever ran them to expect 12, 1 and 1 matches for an anchor in a document before editing it. The real counts were 0, 0 and 0. The numbers had come from memory, or from a reading of a different file. The file was available to the author every time.
The habit: an expected value records how it was obtained, and when: measured, with the date and the tool, or derived from another named value. Writing “measured” next to a number is a small cost, and it makes “I never looked” visible before anyone runs the check.
What they have in common
None of these checks was skipped. Each ran and printed something reassuring. The failures sit in a few shapes:
- a control that routes around the computation it should test;
- a threshold set below the value that matters;
- an error that becomes empty output, which becomes a number;
- a check that reads a place other than the one it assumes.
The defence is mostly a question to ask before trusting any check: what would this print if the thing it guards were broken? If the answer is “the same thing”, it isn’t a check yet. Give it a case that must fail, run that case through the same path as the real one, and say which machine and shell it was tested on, because the same command can mean different things on two of them.