Oops! I Set Up Talos Linux and Elastic Stack On My Homelab

Let's begin with a simple vision: I already had elastic search running in docker for my all-in-one docker compose dev environment. How hard could it be to make a more production-ready (or at least staging) environment?

Three weeks, apparently, of slamming my head against debugging.

It's like weightlifting, but for your mind, using a puzzle that remains difficult the entire time. That's why I love/hate operations work, you have to be enough of a generalist and specialist at the same time to solve problems.

Remember in the previous post when I said that I'm targeting Proxmox with VMs and docker containers, instead of complicating things with Kubernetes? Well, it turns out that a little extra complexity of k8s is actually helpful for getting ecosystem benefits, such as Helm charts to easily deploy the Elastic stack. And this was all enabled by the new-ish Talos Linux OS.

In that previous post, I mentioned that refinements would be coming with time. One such refinement is the addition of Open ToFu (the Terraform fork) to my repertoire of tools. It makes bulk spinning up and destroying VMs a breeze, thanks to the Proxmox provider; and it's more reliable than Ansible alone.

Don't worry, I'm still using and loving Ansible!

I did, however, find this incredible blog post that details how to provision Talos with Open ToFu and Proxmox:

Talos Kubernetes on Proxmox using OpenTofu

Talos is an immutable operating system designed to only run Kubernetes. The advantage of Talos is an out-of-the-box Kubernetes install, as well as a smaller attack surface, and easier maintenance.

StonegardenVegard S. Hagen

With this inspiration, I managed to get my own Kubernetes cluster running on my homelab. There were a lot of problems to troubleshoot, though, including getting OSI Layer 2 ARP announcements working, and allowing my ELK stack to have an ingress with TLS pass-thru.

So yes, it was a pain to learn. Yet it did simplify the process of installing third-party software, like elastic stack. Helm charts are so easy to use, especially since version 3 of Helm — when I started using k8s tooling, Helm 2 was still problematic. It's neat to see the world getting better tools with time.

What's next? Well, I have a lot of infrastructure laid out now. I have growing familiarity with my tools, like ToFu.

Now I need to set up SSO with Keycloak, email with some provider, and I have already set up a public IP forward-proxying HTTPS traffic into my homelab. Once those systems are in place, I can finally return to my game dev tasks, instead of toiling on infrastructure.