Over-engineering my homelab: GitOps in the closet

For the last several years, I’ve ran my homelab on a single Debian node, with a big set of Docker Compose files to define my various services. This is a perfectly sufficient deployment method for small-scale projects like this, but it can be delicate and tedious to edit, and it doesn’t provide much insight into how things are actually run in production environments. Since I’m looking to hone my DevOps and system administration skills, I decided that a rebuild was in order - I wanted to manage my homelab the same way I would manage a production service at scale, with all of the tooling that makes that possible.

The biggest shift is moving from Docker Compose to Kubernetes. We’re running with the big boys now - Kubernetes is the industry standard cloud orchestration platform, and it’s very valuable to learn for someone looking to go into cloud engineering, or really any sort of modern platform engineering. It does bring with it a significant increase in complexity, however, but by front-loading the development of the surrounding infrastructure, a lot of that complexity can be automated away and abstracted out of my daily operations.

This transition has allowed me to improve on a number of aspects of my previous setup:

• Orchestration: Kubernetes for the majority of services, Docker Compose only for experiments
• Deployment: CD tools like Flux and Doco-CD instead of running Docker commands over SSH
• VM/Firewall State: Terraform state management instead of manual changes in the Proxmox UI
• Host management: Ansible for automatic bootstrapping and updates
• Secrets: sopssecrets are tracked in git instead of only living on the hosts
• Rollback: just a git revert for any changes to the cluster or compose stacks

Using GitOps

The goal of this redesign is to centralize everything pertaining to the configuration and deployment of my homelab in one git repository, and use that as the single source of truth for the state of the services and machines - a practice known as “GitOps.” It’s a modern take on the DevOps pipeline that builds around git, where the only way to modify the state of the infrastructure is through automated deploy pipelines triggered by changes to the git remote.

One of the perks of GitOps is that it simplifies the deployment flow dramatically through automation. Say I want to change something in a Kubernetes manifest:

diff --git a/kubernetes/apps/beszel-agent/app.yaml b/kubernetes/apps/beszel-agent/app.yaml
index e1fa7f4..ff4f023 100644
--- a/kubernetes/apps/beszel-agent/app.yaml
+++ b/kubernetes/apps/beszel-agent/app.yaml
@@ -48,7 +48,7 @@ spec:
               value: America/New_York
           ports:
             - name: agent
-              containerPort: 45876
+              containerPort: 12345
               protocol: TCP
           volumeMounts:
             - name: data

This would change the port that beszel-agent (a lightweight system monitor) listens on. I would need to make a few other changes for this to actually work, but assume I already have.

If I were manually administering the cluster, this would be applied with:

kubectl -n private apply -f kubernetes/apps/beszel-agent/app.yaml

However, this would introduce configuration drift: the state of the code on the remote is now different than the state of the cluster. This can get out of hand quickly, and it did pretty often when I was manually editing Docker Compose files on different hosts, so it’s best to avoid using kubectl in this way.¹

With GitOps, pushing the change to the cluster is a bit more indirect, but much simpler:

git add kubernetes/apps/beszel-agent/app.yaml
git commit -m "k8s: changed containerPort for beszel-agent"
git push

Like it says in the name - you just use git for everything. These changes are eventually merged into main, where my continuous deployment tools (Flux for Kubernetes, Doco-CD for Docker Compose) pull down the changes and apply them. Any difference between the repository and the deployment is automatically reconciled - the desired state is the committed state. If the release fails, I can inspect the logs and revert the commit, and the controller will reconcile back to the prior known-good state.

Ultimately, it’s difficult to manage absolutely everything this way. I wound up with a layered architecture that enforces infrastructure-as-code (IaC) practices for things VM bootstrapping and firewall configuration, while maintaining proper GitOps for app deployments - changes to main are applied automatically, rollback is done via git revert, and nothing is ever edited by hand on the hosts.

Docker Compose vs. Kubernetes

Docker Compose is popular in the homelab community for its simplicity and efficiency. It’s very easy to learn, and is perfect for small scale deployments and hobby servers. I enjoy using it, and practically it is the better choice for my homelab - but my goal with this setup is to learn production-grade, industry standard tools, of which Docker Compose is not any longer.

Kubernetes runs the show for production container orchestration, and is significantly more complex to deploy and maintain than Docker Compose. I am fairly new to Kubernetes, and while I have enjoyed learning how the cluster components like the API server, scheduler and etcd interact, I appreciate how k3s abstracts some of these interactions away and lets me focus on learning how to drive the cluster.

Due to the added complexity of running Kubernetes, I’m taking my time and emphasizing stability and maintainability over rapid deployment. I have moved almost all of the services I run into the cluster, but there are some holdouts (like the edge routing stack) that I’m still evaluating options for. What will likely never move out of Docker Compose are projects that I compile myself - anything that I’m developing or testing that I want to run in my lab will stay as a Compose service.

Terraform

Terraform² always interested me but I never had a really pressing use case for it before - it was nice to provision VMs with, but I rarely changed those provisions, and the Proxmox UI is much simpler than most cloud consoles, so it felt unnecessary to add the extra steps. It became more useful when I started adding Proxmox firewall rules and dealing with more intricate VM configurations - now that there’s a centralized source of truth for the state of the hypervisor, it’s much easier to debug.

I currently have four primary VMs - an edge router, my k3s node, a Docker host and a Home Assistant OS VM. I like having the HAOS deployment separate since it makes handling external hardware devices really simple, and it seamlessly integrates with the standard suite of services that go with Home Assistant. This also makes it much less likely to go down when I’m messing with config files.

My firewall rules are pretty simple - least privilege rules for everything with a default drop rule on all hosts. I allow SSH from machines on my trusted VLAN only, and traffic on application hosts is only allowed on specific ports, from the edge VM.

Ansible

I have found Ansible to be a bit clunky in the past. I feel like the ROI crossover doesn’t happen until you have significantly more hosts to manage, and for my relatively small pool of hosts I do miss a certain amount of visual feedback. It has always felt overbuilt for my purposes - but, now that the name of the game is overbuilding for the sake of knowledge, I have automated everything I need to bootstrap and update my VMs with Ansible.

For this homelab, I maintain bootstrap and update playbooks. The bootstrap playbook installs important packages and tidies up some things that my VM template doesn’t catch, and installs either Docker or k3s depending on the role. For Docker hosts, it deploys the Doco-CD service as well. The update playbook just runs a safe system upgrade and reboots if necessary.

Before I found Doco-CD, I also deployed my compose stacks via Ansible, which it was not well suited for. Rendering the YAML and handling secrets required significant templating and variable injection, which was way too complicated for what it was actually accomplishing. Now that my usage of Ansible is more narrowly scoped, I have a better time using it.

Networking

I chose Traefik and Authelia for my reverse proxy and auth provider. Traefik is notable for being supported in both Docker Compose and Kubernetes, and Authelia is the most straightforward auth provider I could find - it is plenty for my needs. The edge routing stack currently lives off cluster, more on that later.

The Kubernetes cluster has its own Traefik instance to handle routing for cluster services. I deploy it as a Flux-managed HelmRelease, instead of using k3s’ bundled Traefik instance, but I do still use the bundled ServiceLB to provide LoadBalancer IPs.³

Using Gateway

I use the new Gateway API everywhere, with one shared Gateway in the traefik namespace - every app attaches an HTTPRoute so that ingress is declarative per app. The Gateway API is the modern routing API scheduled to replace Ingress, so I figured this would be a good chance to learn it. Unlike Ingress, Gateway divides controller-specific behavior and path-to-service mapping into separate resources - the controller has Gateway and GatewayClass manifests to define listener ports, TLS, and namespaces, so my applications just need an HTTPRoute:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: paperless
  namespace: private
spec:
  parentRefs:
    - name: traefik # refers to the Gateway, which refers to a GatewayClass
      namespace: traefik
  hostnames:
    - paperless.example.com # hostname to attach to this service
  rules:
    - backendRefs:
        - name: paperless-service # name of the service to route to
          port: 8000

This way, each application only needs to own the hostname, backend service and port number. I like this because it makes it much simpler to change network controllers if I ever want to - there are far fewer file changes necessary.

Private and public access

Private services are accessible only via Tailscale. Tailscale is a very well designed service, it makes VPN access to my infrastructure very simple and straightforward, and is way easier to set up than the site-to-site Wireguard setups I used to use. I run the VPN client on all my personal devices, and it gives me secure access to internal and personal services from anywhere.

For services that need to be accessible from the internet, I use cloudflared as an ingress point and route its traffic directly to the edge Traefik. I use this so that I can take advantage of their sophisticated WAF platform to reduce unwanted load on my servers. Because all public traffic is forwarded directly to Traefik and Authelia, the routing and authentication are centralized in one place and I don’t need to manage public routes separately.

The two service tiers are in different DNS namespaces, so I can block any traffic to the private subdomain coming from the public internet at the Cloudflare level. I have DNS records in my Tailnet to resolve the private subdomain to my edge VM, so only devices that are connected to Tailscale will be able to reach those private services.

Why edge routing is off-cluster

I chose to keep the edge router in Docker Compose for now. This is more of a blast-radius decision than anything, as it is a dependency for accessing a variety of other things outside of the cluster, as well as tools related to debugging the cluster. By keeping it separate, I can minimize the impact of a cluster or application stack failure, and ensure that services that other people use aren’t significantly impacted - I don’t mind if my log provider goes down if I’m working on things, but if people can’t turn on the AC in Home Assistant then I’ve got a bigger problem.

Network overview

I’ve been building a very simple Mermaid chart SSR in my site generator, so here’s a chance for me to show it off. This chart shows the network flow of requests for both public and private users accessing services in my homelab:

Storage

Since my cluster is single node and Docker services don’t handle any significant persistent data, I am using the local-path provisioner that comes with k3s. I provisioned a separate volume in Proxmox for container storage and back it up with the VM via Proxmox Backup Server.

I’m not super trusting of my ability to not accidentally delete things, so I give all PVCs a slightly modified StorageClass:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path-retain
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: rancher.io/local-path
reclaimPolicy: Retain # important: deleting a PVC does not delete the data
volumeBindingMode: WaitForFirstConsumer

I have had to rebuild deployments a couple times while learning the ropes, and implementing this made restoring my application data much easier. The pruning behavior of Flux is constrained around live data, so if I do remove application manifests it won’t unintentionally garbage-collect my data.

Hopefully, when it’s financially feasible, I’d like to build a dedicated storage node, which I will then probably connect to a more sophisticated storage controller. For now, this works fine.

Guardrails

In order to make sure that everything in my homelab stays secure, organized and up to date, I use a number of Github Actions to enforce code quality and security. For starters, I run a couple linting tools: kubeconform for Kubernetes manifests, yamllint and ansible-lint for YAML files, and shellcheck for shell scripts. While I have all these tools attached to my editor, sometimes things get through that I didn’t catch that would otherwise cause problems, so I need to have a security barrier on the remote before my CD pulls the changes down.

I also run Renovate to keep my container images up to date. It updates the container tags on a schedule and submits PRs with updated versions, which I can then merge to update the containers. This is a new feature that I’m still testing, but it seems much more robust than my previous usage of Watchtower, which didn’t have any verification process for updates.

Handling secrets

It’s a bad idea to check API keys, passwords and credentials into version control - even if the repository is private, it’s stored in git history unless you explicitly remove it, so if anyone gets access to the repo, they’ll have your sensitive information. If we are to track everything about the infrastructure in git, we need a way to encrypt this information before storing it.

SOPS is an elegant solution to this problem: it asymmetrically encrypts your sensitive files in a structure-aware manner: in YAML files, for example, it only encrypts the values, not the keys. The public key is stored in git, and the private key is imported into the Flux and Doco-CD runtimes, as well as being stored locally (and backed up!) for editing purposes.⁴ This allows me to track changes to sensitive data without actually tracking the plaintext data itself.

In case I left any secrets lying around in plaintext, I run an idiot check action to make sure I’m not committing anything that should be encrypted:

check_file() {
  path=$1
  # .sops.yaml is the sops config itself, never encrypted
  case "$path" in
  .sops.yaml | .sops.yaml/*) return 0 ;;
  *.sops.yaml | *.sops.yml | *.sops.json | *.sops.csv | *.enc.yaml | *.enc.yml | *.enc.json) ;;
  *) return 0 ;;
  esac
  # sops filestatus: "encrypted": true means the file is in encrypted form
  status=$(sops filestatus "$path" 2>/dev/null) || {
    printf 'unencrypted sops file: %s\n' "$path"
    fail=1
    return
  }
  encrypted=$(printf '%s' "$status" | python3 -c \
    'import json,sys; d=json.load(sys.stdin); print(d.get("encrypted", d.get("Encrypted")))' \
    2>/dev/null) || encrypted=False
  [ "$encrypted" = "True" ] || {
    printf 'unencrypted sops file: %s\n' "$path"
    fail=1
  }
}

This runs as a pre-commit git hook with my private key, and runs keyless in CI as an independent enforcement point.

Conclusion

As I stated, not everything can be GitOps, nor should it be. Git-managed state is perfect for Kubernetes and Compose, and Terraform and Ansible, while not strictly GitOps, are consistent and stable - as long as the tools are documented or self-documenting, they’re just as effective as the GitOps controllers.

The goal of this homelab is not necessarily to make my tiny homelab work like AWS - it’s an effort to make changes to my infrastructure more reviewable and recoverable, so that I have the freedom to learn the tools that are used on similar enterprise platforms. I think it accomplishes that goal pretty well.

I don’t have the repository public right now, since I don’t want to make any sensitive data publicly viewable, even in encrypted form. I’m working on a sanitizer that will generate a public version of the repo for demonstration purposes.