In this series

  1. Why I modernized my homelab instead of upgrading it
  2. Making a homelab repository agent-friendly
  3. Building the new K3s foundation
  4. Proving the entire platform with one blog
  5. Handing PostgreSQL over to Amp
  6. Modernizing an application, not merely moving it
  7. Replacing a surveillance media chain, not just a recorder
  8. Proving Frigate before the cutover
  9. Upgrading observability beyond the old cluster (this post)
A developer reviews dashboards inside the homelab while a separate monitor watches the platform from outside Kubernetes

The new stack explains what is happening inside Kubernetes. A separate host checks whether the platform can still be reached from outside it.

I first set up monitoring for my homelab about five years ago. It could still draw graphs and send alerts, but the stack had become old together with the cluster it watched. Five years is a long time in software. Some of the components were almost ready for their own retirement post!

The replacement itself was not very exciting. I now use VictoriaMetrics, Grafana, Blackbox Exporter, vmalert, Alertmanager, Loki and Alloy. It is a modern, fairly standard observability stack, and there is little value in explaining every component yet again.

The interesting part is what happens after the stack finds a problem. In the old setup, an alert waited for me to notice it, understand it and find enough time to fix it. Alerts are very patient. Production problems are usually less so.

Now a selected alert can wake an Amp Orb and start the investigation by itself. The homelab is no longer only monitored continuously. It can also begin triaging a problem, preparing a fix and moving it towards deployment while I am away.

Let an alert wake an Orb

The setup is quite small. An Amp plugin gives me a private webhook URL. I store it in a Kubernetes Secret, then configure Alertmanager to send selected critical alerts to it. There is no extra incident platform in the middle. Alertmanager simply talks to Amp.

When a webhook arrives, the plugin checks the payload, keeps only the useful facts and ignores duplicate retries. It wakes a medium-mode Orb for the incident. If the same problem returns, the new alert goes into the same thread, so the Orb keeps the earlier context instead of starting from zero. I fired test alerts through the complete path to make sure this actually worked.

The handoff is fully agentic. Nobody has to notice the alert and copy it into a prompt. The Orb begins with read-only checks, looks at the evidence and decides whether the problem needs a code or configuration change. If it does, the work can continue through the same repository, beta checks and deployment path from the earlier posts.

This does not mean an alert has permission to change production. That still stops for my approval, which is probably for the best. The game changer is that the alert can start the work, keep the context and move towards a checked fix before I even open my laptop.

Keep one witness outside Kubernetes

An external watchdog is not a new idea. I always knew I should have one, but I could never be bothered to set it up. There was usually something more interesting to build than a script that checked whether the other scripts were alive.

Amp finally took care of it for me. The watchdog runs on the free-tier Oracle VM where I already run OpenClaw and checks the public paths every minute from outside my homelab. It is simple, independent and no longer sitting on my to-do list.

A rebuilt and agent-managed cluster

After all this work, I now have a rebuilt cluster with old services replaced by newer stacks and components. More importantly, the whole platform is agent-friendly, agent-assisted and increasingly agent-managed.

This is not vibe coding. I did not ask an LLM to make the homelab “modern” and hope for the best. A lot of thought went into where things belong, how changes are tested, how beta stays separate from production, how deployments stop when checks fail, how the platform is watched and how the next service can be added without making a mess.

These are simply good engineering practices. The difference is that agents can now follow them even when I am not sitting at my desk. The cluster is not fully autonomous, nor do I want it to be. It can investigate, fix and deploy through a path designed for agents, while the important decisions still come back to me. That is a much more useful kind of automation.