I recently stood up a 42U server rack in my basement. Some people are into cars, I’m into servers. 😁
It’s a bit empty right now, but with plenty of room to expand for future projects! All the CAT6A cables in my house terminate at the patch panel at the top of the rack, with a very basic router / switch setup that plumbs it all together.
The thing I’m more excited about is the 3 node MINISFORUM UM750L mini PC cluster that I’ve started building on top of.
I’m programming a custom VM orchestration and control plane on the cluster called “lightbyte”. Currently, the system supports:
- Dynamically provisioning and placing Firecracker virtual machines on the worker nodes.
- Provisioning block storage volumes from rootfs images. These images are built via Nix, and automatically get deployed to the cluster during software updates.
- Scale groups, for dynamically scaling up multiple identical VMs. Sort of like EC2 autoscaling groups. Way overkill for a homelab, but why not?
- Rudimentary HTTP load balancer / proxy support via pingora. These can target scale groups or individual sets of VMs.
- Worker draining, to vacate all resources from a host node
Here I create a basic scale group with a configured rootfs image, then scale it up to 5 nodes.
Provisioning is all done via reconciliation controllers in the control plane server processes. Each resource has its own separate controller that’s responsible for reconciliating desired resource state (stored across all 3 etcd nodes), against the current state of running resources on each worker.
Once VMs boot, they bridge to the physical network via a Linux bridge.
HTTP load balancers run inside guest VMs themselves, owned and managed by their own scale group. Once I implement DNS in the cluster, my plan is to have DNS records that will round-robin across multiple proxy VMs, which themselves will proxy to other underlying scale groups or collections of VMs. I dynamically push backend topology changes to the proxy guests over a guest vsock socket server. This vsock socket allows the host machine to orchestrate the guest and push configuration changes without the guest needing to talk to the control plane directly.
Here are some of the software details:
- Implemented in Rust. Control plane, worker agents, guest agents, command line clients, etc.
- Control plane database is etcd, running on all 3 nodes, which stores all resource object state. I use etcd transactions to perform transactionally correct read-modify-write on the objects. Reconciliation controllers are leader-elected with etcd and executed on the control plane nodes. I use leader election to make sure there’s only 1 controller of each resource type running at a time.
- Workers all run NixOS. The entire system is deployed and updated with a single command via colmena.
- I’m using firecracker for the Vmm, which sits on top of Linux KVM for the hypervisor.
This has been a fun one to build. I’m looking forward to stabilizing things enough to use the system for all my homelab services. It’s a little baby homelab cloud!