This is written by claude ( I think he’s funnier than me …)
Foreman 3.11 to 3.19 in a Day (or: How I Learned to Stop Worrying and Love the Hop)
Our Foreman server had a problem, and the problem was 2024 calling. It was running Foreman 3.11 — end-of-life for over a year, roughly six releases’ worth of security fixes behind, and precisely the sort of thing that fails a Cyber Essentials “supported software” test. Today we fixed that, and rather more besides.
The rules of the game
Foreman has a simple support policy: only the latest two releases get updates, and with three releases a year, a version goes stale fast. It also has a simple upgrade policy: one release at a time, no skipping. From 3.11 to the current 3.19, that’s eight sequential upgrades — repo bump, package upgrade, installer run, smoke test, repeat.
Each hop looks like this (shown for 3.16 → 3.17; set NN and go):
NN=17
# Rollback point first — snapshot the VM, and dump the DB for belt and braces
sudo -u postgres pg_dump -Fc foreman > /var/backups/foreman-pre-3.${NN}-$(date +%F).dump
# The release RPM carries the new repo definition AND the new GPG signing key.
# It must go first — skip it and the next step fails GPG checks on everything.
dnf upgrade https://yum.theforeman.org/releases/3.${NN}/el9/x86_64/foreman-release.rpm
rpm -q foreman-release # verify before continuing: must say 3.NN
# The plugins repo is NOT updated by the release RPM — bump it by hand
sed -i "s|/plugins/3\.[0-9]*/|/plugins/3.${NN}/|g" /etc/yum.repos.d/foreman-plugins.repo
grep baseurl /etc/yum.repos.d/foreman-plugins.repo
# The upgrade itself. Check the transaction: core packages at 3.NN, and the
# essential plugins (foreman_puppet, foreman-tasks) at the matching fm3_NN
# dist tag. Held-back plugins = missed repo bump = abort and fix.
dnf upgrade
# Runs the database migrations. Must end "Success!" — never hop on a failure.
foreman-installer
# Smoke test before the next hop
curl -sk https://localhost/api/v2/ping | python3 -m json.tool
systemctl --failed
Then increment NN and repeat. Fifteen to thirty minutes per hop on a plain Foreman + Puppet setup.
There was also a boss fight waiting at the mid-point: Foreman 3.17 onwards requires Puppet 8 on the server, and our puppetserver was on Puppet 7 — itself EOL since February 2025. Worse, Puppet’s owner (Perforce) has withdrawn free open-source binaries entirely: no security fixes for us, and no packages at all for Rocky Linux 10. The answer is OpenVox, the community fork maintained by Vox Pupuli — a drop-in replacement for Puppet 8 with the same commands, config paths, and certificates, and the one the Foreman project itself runs on.
The OpenVox server swap, condensed:
tar czf /root/puppetlabs-preswap-$(date +%F).tar.gz /etc/puppetlabs/
systemctl stop puppetserver puppet
dnf install https://yum.voxpupuli.org/openvox8-release-el-9.noarch.rpm
dnf install openvox-server openvox-agent # replaces puppetserver/puppet-agent; CA, certs, config all survive
rm /etc/yum.repos.d/puppet7-release.repo
ls -l /usr/lib/jvm/jre-17 && /usr/lib/jvm/jre-17/bin/java -version # OpenVox 8 wants Java 17 — check first
systemctl start puppetserver
# Teach the Foreman installer the new package names (persists for future upgrades)
foreman-installer \
--puppet-server-package openvox-server \
--puppet-client-package openvox-agent
And on each client node, when its turn comes (or via a Puppet class doing the same):
dnf install https://yum.voxpupuli.org/openvox8-release-el-$(rpm -E %rhel).noarch.rpm
dnf install openvox-agent # replaces puppet-agent in place, certs kept
rpm -q openvox-agent # dist tag should match the OS — el10 on Rocky 10
So the plan became: hop 3.11 → 3.16 on Puppet 7, swap the puppetserver to OpenVox 8, then hop on to 3.19. Our ~200 Puppet 7 agents keep working against an OpenVox 8 server throughout, so no estate-wide flag day.
What actually happened
The hops themselves were pleasingly mechanical. The interruptions were where the day earned its war stories:
The stowaway plugin. A full package audit turned up puppetdb_foreman — a plugin for a PuppetDB we don’t have, still built for Foreman 3.10, quietly along for the ride. Its front-end (a Puppetboard container) turned out to be a zombie too: the systemd unit had been failing at every boot since Docker was removed from the box. Both got a decent burial.
The GPG ambush. Each Foreman release is signed with a new key, delivered by the release RPM. Skip that step (guilty) and the repo serves new packages the old key can’t verify — 29 packages of “GPG check FAILED”. No harm done; nothing installs on a failed check. Release RPM first, then everything else.
The Java excavation. The OpenVox swap needed Java 17, which was “installed” — except a patch run the previous week had been interrupted, leaving ~35 packages with duplicate RPM entries and Java’s alternatives symlinks pointing at deleted files. dnf remove --duplicates sorted it, and gifted us a new estate-wide health check: if a box has duplicate packages, its last patch run lied to you.
The delayed-fuse Apache. That same duplicate cleanup reinstalled PHP (why is PHP on a Foreman server? Archaeology for another day), which dropped a config file into Apache’s conf.d that referenced a module Foreman’s Apache doesn’t load. Nobody noticed — until the post-upgrade reboot, when httpd refused to start. This is exactly why you reboot during the change window rather than discovering these things at 3am some future Tuesday.
The legacy facts. Puppet 8 removes old-style facts like $::fqdn, and treats them as compile errors. Our codebase had exactly two offenders — one user manifest and one MOTD template — both trivially fixed to the modern $facts[] syntax. The MOTD template then provided the day’s best receipts, announcing puppet7.34.0 → puppet8.28.0 in the login banner of every migrated node.
Where we ended up
By mid-afternoon: Foreman 3.19.1 (a supported release for the first time in over a year), OpenVox server 8 with certificates and agents untouched, verified across Rocky 8, 9, and 10 and Debian, over both direct and load-balanced paths. Agent migration to OpenVox is now self-service: a small Puppet module attached to a Foreman host group converts nodes on their next check-in — dev groups first, then wave by wave through the summer patch rounds. Even the Rocky 10 boxes, which had been running sideloaded EL9 agents because official Puppet never shipped EL10 packages, now get native builds.
The takeaways
- Staying current is cheaper than catching up. Eight hops in a day was fine, but twice-a-year single hops would have been trivial. The runbook now exists; future-us has no excuse.
- Upgrades are audits in disguise. The dead plugin, the zombie unit, the half-applied patch run, the PHP landmine — none were caused by the upgrade, all were found by it. A box that’s never exercised accumulates quiet lies.
- Reboot while you’re watching. Two of the day’s failures only manifested from a cold start. Better today than during an incident.
- The Puppet ecosystem moved — follow it. If you’re on Puppet 7 or frozen Puppet 8 packages, OpenVox is the maintained path forward, and the migration is genuinely undramatic.
Total damage: one morning, one afternoon, zero user-facing outages, and a login banner that finally tells the truth.