We will be undergoing a hardware refresh soon for one of our clusters. Ideally we would add new servers into the existing cluster, evict VMs and storage from the old hosts, and then retire the old servers to make it zero downtime. The new servers will be differently spec'd to our old ones. I think NIC naming differences are likely to be a problem for the storage network. For the mgmt network, that is per node via our PXE install config options. We have a separate compute network but we can turn VMs off one by one and change their required networks. For storage, I don't think there is any way to say "use these NICs on some nodes, and these NICs on other nodes", has anyone run into this before? A hacky approach would be rename the new NICs via kernel args to match the old ones but that doesn't seem very nice.

Please be aware of any CPU platform differences between your new hardware and existing hardware. You may have to add custom tags to the new hardware that matches the CPU architecture of the existing hardware for VMs to be able to migrate. These tags would only need to be in place for an interim period until VMs are fully migrated and the current hardware is decommissioned. Ref: github.com/harvester/harvester/issues/7096 Note: I have not upgraded past 1.7.1, and have not read the latest release notes to see if this (cluster with mixed CPU generations) is now handled differently by Harvester or the upstream KubeVirt.

Thanks, will keep an eye out

I'd do some testing, but it's possible to change the defaults to have kubevirt set a standard compatible model across your fleet by default to help with the CPU stuff Zane is bringing up. (an excellent point imho) Typically there are no issues migrating from older to newer but not newer to older, but if you set it to use a compatible cpu type (older) on the newer stuff it can work (but not always depending on intel/amd/etc) I have a script somewhere for looking at nodes for that if you want it.

I'm assuming this is only a problem for live migrate and if we turn off and on each VM it won't be an issue?

yeah but upgrades use the live migrate to complete

We don't plan to do a harv upgrade during the hardware refresh and will keep the same version. Hopefully won't be an issue

Well for the most part you can live migrate from older to newer fwiw.

fingers crossed, thanks!