Talk: Kubernetes

Hello,

I'm setting up automatically a virtual Kubernetes cluster with NixOs. This is a broad topic, and it should be an open discussion. So, I opened it and summarized my questions and findings here.

What I Need:

Defines the cluster architecture: number and type of head and worker nodes.
Defines the VM node's hardware profiles (# of CPUs, # of GB RAM, Disk Space, and network characteristics)
Define the scheduler//resource provisioner stack. In this case, Kubernetes + Volcano, but it could be SLURM, Flux, or others.
Automatically deploy the whole cluster

How I want to build it:

I use Hydra to describe the experiment configuration (cluster architecture, #of nodes, type of scheduler).

Then I use https://github.com/nix-community/nixos-generators to build a NixOs Image with all the software preinstalled but not configured.

Then, I can provision those images to a cloud using OpenTofu or, in my case, a virtual cluster composed of VMs. I use a custom Python program that leverages `libvirt` python API because the `terraform-provider-libvirt` didn't work well for me and is maintained "on a best effort basis."

Finally, I must configure the rest of the stack at runtime (nodes joining the cluster as a worker or head node). This is the part I'm unsure of regarding Nix. Can Nix be used for runtime configuration as well? When I provision the images, how can I configure them using Nix? Should I use another approach like Ansible, bash, or Python?

To Sum Up:

VM Image: Built with Nix (immutable, reproducible)
Provisioning: Custom Python script using libvirt (dynamic VM creation, disk management, IP discovery)
Runtime Configuration: Currently unclear. I need to dynamically configure the Kubernetes cluster once the VMs are running.

What are your recommendations for getting this cluster up and running? Should I use a custom Python script to handle the runtime configuration that will ssh and configure preinstalled Kubernetes? Preinstall Kubernetes on the VM with Nix Generators. But how can I configure the network as the configuration requests the head node masterIP parameter?

Here are some references that helped me: