Good morning. We are running some bare-metal k8s clusters using self-built elemental images based on debian, fitted to work with our underlying bgp fabric. The same lifecycle, fully managed in rancher. I was wondering if we could switch to harvester keeping our images (or adapting them for harvester) and network structure? We want to run virtual clusters using kubevirt, but with rook-ceph instead of longhorn. AI said no, custom images with harvester is not possible, I'd like to be sure before we assemble our own puzzle.

You can use SL Micro which is the foundation. Not sure why the OS choice matters for you. Yes, you can run k3k on Harvester via rancher or manually provisioned. Yes you can use rook-Ceph, but Longhorn is a requirement yet due to how it’s built, but you don't need to use it. We enable options to decrease the footprint, as we figure out when we'd fully decouple.

We've had some issues in the past with SL Micro, potentially caused by knowledge gaps on suse products. Mainly while trying to install additional dependencies like frr and its config, and our network doesn't support dhcp. Our current build pipeline works flawless, offering us repeatable, immutable images - something I never managed to do with suse's images.

you can sorta dissect/disassemble harvester and only use the pieces you like, but at some point you're just doing kubevirt anyway basically

Yup, that's our plan-b. It would still be nice to have the turtles/capi integration to let rancher handle the cluster lifecycle, and we also need the kubevirt-csi and the ccm to handle lb services. I was just looking for the most simple option - something like an image-option in a cluster-resource and a disable-longhorn flag, instead of assembling the parts on our own. We just finished a PoC today deploying a k3s cluster in kubevirt and having it survive live migrations.

@chilly-agency-73377 It sounds interesting, do you have a high level architecture of your cluster? or ask AI to make one, for better understanding, thanks.

Sure, in short: • rancher on managed k8s with a cloud provider + elemental operator (stage 0, provisioned via terraform) + argocd for workload deployment to every rancher downstream cluster • local datacenter with 3 availability zones, 1 rack per zone. • 2 ToR-switches per Rack in a mesh running unnumbered bgp, connected to provider fabric for public IPs • bare-metal servers running custom elemental images prepared by operator • images containing script generating identity based on lldp port-identifiers, used to set up dummy fabric interface with calculated ip (encoding zone, rack and unit) and hostname, starts frr for fabric connection, allows elemental-system-agent to connect to rancher • rancher installs rke2 cluster with cilium (nativeRouting, kube-proxy-replacement, multiPool ipam, DSR, bgp-controlplane via host-frr), topology-aware -> called "infra-cluster" • rook-ceph for datacenter-scoped storage (cross-zone availability, zone as failure-domain -> data replicated across all 3 zones) Currently in PoC (promising): • KubeVirt on infra-cluster • Rancher using CAPK to deploy workload-cluster-nodes (using kubeconfig for infra-cluster) • Rancher provisioning uses generated kubevirt instances on infra-cluster to create Cluster, full lifecycle capability established (we might switch to cnpg/kine as external datastores to avoid issues by changing pod-ips) • kubevirt-csi or ceph-csi-direct for rbd block device creation and attachment to kubevirt VMs for use in workload clusters, 1:1 mapping (instead of one large PV containing everything) • cloud-provider-kubevirt to generate LB services on infra-cluster for every virtual LB service (enabling use of public IPs in virtual clusters including status-sync) So far the full chain has been proven. Live-migration of virtual nodes on host-drain works, <1s interruption inside virtual clusters. Scaling and reconfiguration of workload-clusters works. Not sure if that's the intended way, but it seems to work. A combination of CAPK+CAPRKE2 did work as well, but required exposition of the virtual cluster control-plane - we don't like that.

We have a few clusters where we're mixing Elemental running on bare metal and Elemental running on segregated Harvester clusters all running on the same cluster. We're using the Elemental plugin from Rancher to make our elemental images, but you can absolutely bring your own custom cloud-init based images for Harvester. As long as the elemental stuff gets started properly, it'll check in and register the VM as a machine in the machine pool and get assigned to clusters like any of the bare metal nodes.

Sounds good - for us it would require moving everything we currently do in dockerfiles into cloud-init. And we'd need to make sure that things get applied at the correct stages. And we'd need to bake some of the cloud-init parts into the images so they can boot with the correct network config. Basically a lot of work to get close to where we are at the moment 🤣 (For comparison: the only thing we do via cloud-init at the moment is setting up user accounts and ssh keys)

Actually if you have a Containerfile, I think you can do container images on the VM side too.

In Kubevirt? Sure, that's an option. But so far all the fixes we had to add during the current tests were all done using the crd-provided options - fully running on standard ubuntu lts cloud-images.

Ah, I think I understand what I'm getting confused about, maybe you can clear it up for me: > We are running some bare-metal k8s clusters using self-built elemental images [...] I was wondering if we could switch to harvester keeping our images (or adapting them for harvester) and network structure? Are you talking about converting those nodes to Harvester Nodes, or or are you talking about running those images IN Harvester?

AI said no, custom images with harvester is not possible, I'd like to be sure before we assemble our own puzzle.

back to initial question, does it mean a

virtual machine

image in Harvester's concept ?

Oh, sorry about that! We have running elemental clusters and we're very happy with them. We're trying to deploy workload clusters (using harvesters term) inside our elemental clusters. Migrating to harvester-based infra-clusters is no option for us, our network structure isn't compatible with the provided images. If we can use our elemental-images (with adaptions) instead of the provided images, moving to harvester would be an option. If we can get the harvester functionality (the part about deploying workload clusters onto harvester clusters) without needing to run the provided images on our hardware, we'd be happy - the part that allows defining a workload cluster in rancher followed by the deployment and setup of the kubevirt-vms on the infra-cluster.

Ah I think the answer to this is probably "No" - while all things are possible there's nothing to hook into like this for Harvester and run the platform (kube-virt) without using the supported stack.

That was the confusing part - what method does rancher+harvester use to get the workload vms running on downstream clusters. We assumed it uses some combination of CAPI-Providers.

Harvester upgrades and stuff wants to manage the entire OS layer too. It's opinionated, and while you can manage/amend/extend feature sets with things like the network and storage layers, it's not designed to rip/replace the OS layer like how Elemental does it, nor install on top of any kube cluster like how Rancher does it.

We're using that provider, as capi provider. That gives us the VMs on the infra cluster.

Yeah, I'm 99% sure that only the stack in the official iso releases are supported for official #C01GKHKAG0K and nothing else. That's different from #C028DVCAYLD nodes or using other providers with #C3ASABBD1 rancher stuff. But I don't officially work for Suse, so maybe there's another answer.

At this point it's more or less just curiosity. We're only testing things like using an external datastore and the lifecycle mechanics before we put it into our automation - the result is important, not necessary how we got there. Claude originally tried the setup using the same kubevirt cloud provider, but with the rke2 capi provisioner afterwards, but that required the ability for rancher to access these VMs using ssh, so inverted direction compared to elemental. Maybe we did it wrong.

You might be able to package your own OS by dissecting the harvester-installer repo which uses elemental-toolkit under the hood. Also, part of what you're looking for for downstreams is CAPHV which isn't quite GA yet.

That operator was the missing part, thanks! We'll figure out if it depends on an actual harvester installation or if it runs on our (close to harvester) setup as well. I really appreciate the work harvester does! We simply spent 6 months already on our infrastructure (a big part of that in the network) so scrapping that and moving to a new platform is no option, with a tight deadline ticking closer. Our current plan is to keep an eye on the harvester development considering future changes to our setup, so that we might be able to switch over at some point without too much interruption. CAPK and the rancher provisioning is very similar, making that switch later easier.

Yea we absolutely prefer the rancher/SUSE stack over others for the ability to break down or assemble and customize the stack. Lots of great things otw from sneak peeks. Also keep an eye out for elemental 3 as well.

elemental 3 as in rancher/elemental or suse/elemental?

rancher/elemental is 2, suse/elemental is 3

Any idea if suse/elemental still works with the rancher/elemental-operator? That connection is why we used rancher/elemental so far.

Yea, there will be an upgrade path but I don't know if 3 has backwards compat yet. There are some fundamental changes that required breaking things. So I'd say don't upgrade yet but keep an eye out.

Will do, thanks! I love that operator. Deploying images ~2 times per week for half a year now, not a single issue. No need to touch nodes directly.

Same here. And we're automating harvester host/cluster provisioning with it too, though we have to supplement with some other stuff. But we're working on moving that to 3

Will do, if I remember 😄 We created a nice little deployment pipeline using rancher, argocd and custom tooling for reusable deployment templates across multiple clusters. Everything automated. Including deployments touching multiple clusters, using fleet resources. Like our offsite backup for s3 buckets, where the mgmt cluster creates s3 credentials using a service account with the cloud provider, distributing it down to the infra clusters into the ceph rgw deployments for multi-zone replication. So many possibilities.

I think that's by design or CAPI spec, also might be a backlog item. You could prob establish a tunnel from downstream to upstream then set that as the lb. We do something similar with tailscale sometimes but haven't tried it specific to a workaround for capi

so is suse elemental no longer distro agnostic? it sounds conceptually different than v2

Elemental 3 provides a SLES-based, immutable, image-oriented Kubernetes-native infrastructure stack

I doubt there is intent to lock it down

yeah I just wish there could be some effort at an agreed standard between elemental/bootc/etc. positioning it as a SLES distro seems to get further from that

In a perfect world... I'd love to see a nixos-built elemental image, but I'm afraid that too far off any popular standard to become reality at any point 😄

Yea it's the great supportability conundrum