My home server and the small cloud machines around it all run Ubuntu. Keeping them safe comes down to two jobs: knowing which machines have a known vulnerability with a fix waiting, and patching them the same way every time. Daily checks do the first and one script does the second. Both live in the homelab platform‘s repository.
The problem
Every machine needs package updates, and some updates need a reboot. Done by hand, each host is a run of small decisions: is anything running that a reboot would interrupt, did every service come back, is this the right moment. Without a check, I’d only learn that a machine was exposed when I happened to look. Ubuntu Pro’s own tool can answer that per host, but parsing Canonical’s vulnerability feed takes about 900 MB, more than the smallest machines have.
What I built
- A daily CVE check. Once a day,
cscollects every host’s package list over SSH, downloads Canonical’s feed and runs Ubuntu Pro’s own parser over each list. Every CVE is then checked against CISA’s Known Exploited Vulnerabilities catalogue. One Uptime Kuma row carries the verdict, and a red row reaches my phone through ntfy. The check installs nothing and reboots nothing. - A daily image check. Since 22 October, a second check reads the packages inside every running container image on every server.
- One patching script, run on demand. It runs from my workstation, never from a timer, and works through the hosts one at a time.
- A reboot alarm says when a reboot is owed, and the script is how it gets done.
How it works
The CVE check gives each pair of package and CVE exactly one class: a kernel package that isn’t the running kernel, a fix that can be installed, a fix only in Ubuntu Pro’s extended archive, a fix the host already has that the feed can’t map to a package, or no fix published yet. Anything the feed doesn’t explain is a fault, and that host reads as unreachable, never as clean. A host turns red when an installable fix of high or critical priority is waiting, or when a CVE is on CISA’s list.
The image check builds a software bill of materials per image with Syft and checks it with Grype and CISA’s list: a listed match, or a critical one with a published fix, turns the image red.
The patching script goes smallest blast radius first and cs last. On each host it runs apt-get update and an upgrade that keeps existing configuration files, and reboots only where one is needed. Then it checks the host’s health: a new boot if it rebooted, the system running, every always-on container and enabled service that ran before running again, and no unit newly failed. A service that systemd had to restart by itself during the run is reported, not hidden. A host that fails after something changed stops the whole run, and the hosts after it are reported as not started.
Before touching a host, seven guards can refuse it and change nothing: no passwordless sudo; the host is the machine the script runs on; a stopped container that a reboot would start; a hold placed on a container on cs; the docs publishing lock held on wp; a nightly backup running; or a container stopped by hand that a reboot would leave down.
Decisions and trade-offs
- Parse the feed on
cs, not on each host. Running Ubuntu Pro’s tool everywhere is the simple path, but it needs more memory than the smallest machines have, and it spiked the others every morning. The cost is thatcshas to reach every host, so an unreachable host reads as unreachable rather than clean. - Prove the check still agrees with the vendor. Every run that downloads the feed also runs Ubuntu Pro’s unmodified path over
cs‘s own packages and compares it with my result byte for byte. If they differ, no host is checked. I gave up a check that always answers for one that says when it can’t be trusted. - On demand, never unattended. Unattended upgrades would patch sooner. I start every run myself, so a reboot happens when I’m there to read the health checks.
wponly when named. A bare run skipswp, because its reboot takes all four WordPress sites down at once.
What’s next
Nothing here patches by itself, deliberately: the reboot alarm and the CVE rows say when, and I decide. The cost is speed: a fix lands only as fast as I act after a row turns red.
Built with
Ubuntu, Ubuntu Pro client, CISA’s Known Exploited Vulnerabilities catalogue, Syft, Grype, Bash, Python, systemd timers, Uptime Kuma, ntfy.