Homelab, rebuilt — Part III: Building the new K3s foundation

In this series

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation (this post)

K3s itself was not new to me. For my old cluster, I followed Jeff Geerling’s Raspberry Pi cluster guide and customized the K3s Ansible playbooks it recommended. That got me running quickly, but over time my version of the setup and the system it created became harder to reason about.

This time I wanted to try something different. Instead of finding another setup and slowly turning it into my setup, I described the outcome and let Amp work out how to get there.

The hardware did not change. My four Raspberry Pis stayed exactly where they were; I only reinstalled Raspberry Pi OS from scratch. The new part was one Ubuntu Server VM running in Hyper-V on Windows 11. It would run the K3s control plane, work as an AMD64 node and welcome the Pis back as ARM64 workers.

Keep the control plane boring

A single server is not highly available. I knew that. However, extra control-plane nodes would add more moving parts while I was still trying to understand the basic setup. For now, I wanted the control plane to be simple, reproducible and boring—and I mean boring as a compliment!

K3s uses its default SQLite datastore. I added resource reservations to protect the control plane, then disabled the bundled Traefik and ServiceLB because I wanted GitOps to manage those parts later. This gave me a small foundation without pretending it was already the finished platform.

Turn the plan into something repeatable

I had already written the rebuild plan, so I told Amp: “Create scripts to prepare this host as the kube control plane according to the rebuild plan.” On my local Amp runner, it turned that plan into scripts for networking, security updates, SSH, firewall rules, secrets encryption and K3s. I ran those scripts manually on the real VM and pasted the failures back into the thread for Amp to fix.

Before I ran anything, I asked Oracle to review the changes. This was foundation code. Every application would eventually depend on it, so an extra review felt essential rather than excessive. The review pushed me towards the official K3s installer and CLI wherever possible. Amp could still customize the official script with my pinned version and options; it did not need to reinvent the installer in shell. K3s already knows how to install K3s (and future Vincent does not need another clever script to debug!).

The scripts also test key-based SSH before disabling password access. That is a small check, but a very useful one. Automation should prove the replacement works before removing the old access path. Opening the Hyper-V console because I locked myself out would not be a fun start.

The first run broke DNS!

The first run quickly found two bad assumptions. One firewall command tried to insert rule number one into an empty firewall, where rule number one did not exist. More importantly, the default-deny firewall blocked the Pi-hole resolver already running on the VM. Kubernetes looked healthy, but household DNS was broken!

I told Amp exactly what I could see: “Now the Pi-hole instance on this host is not responding to DNS queries.” It traced the problem, made the firewall commands safe to run repeatedly and restored DNS from the LAN. Then it added DNS to the verification so the same mistake could not quietly return.

This felt very similar to the time Amp followed one wrong DHCP lease all the way to a powerline extender. I did not need to guess the fix. I described the symptom, gave it the context and let it follow the evidence. A healthy Kubernetes node was not enough. The same VM also served my home network.

Add backups before moving on

Because this cluster uses SQLite, K3s’s etcd snapshot command does not apply. My backup briefly stops K3s, creates a consistent archive, starts K3s again and transfers the compressed result to the NAS.

There is one important trap here. If the NAS is not mounted, its usual path still looks like an ordinary directory. A careless backup job could happily report success while filling the VM’s own disk! Mine stops unless it can prove the NAS is mounted, then writes a checksum beside every archive and removes old backups according to the retention policy.

I did not test a restore yet because there was still nothing meaningful to restore. The cluster had no workloads or important state, so restoring an empty foundation would prove very little. That test will matter once real applications arrive.

For now, I had a clean K3s cluster, repeatable setup and backups ready before there was anything precious to lose. Better still, the first failure had already made the checks stronger. It was time to build the rest of the platform.