Hey all we have been able to live migrate VMs which have rep Rancher Users #harvester

Hey all, we have been able to live migrate VMs whi...

brainy-kilobyte-33711

05/02/2025, 8:59 AM

Hey all, we have been able to live migrate VMs which have replica count 1 storage attached to them. We were under the false impression that wasn't possible due to the harvester upgrade checker blocking upgrades when VMs with replica 1 storage are running. Please can someone explain why the upgrade checker will block upgrades when we can live migrate these machines ourselves? related issues https://github.com/harvester/harvester/issues/7323,

brainy-kilobyte-33711

05/02/2025, 9:08 AM

The live migration moved the replica to another node which I suspect isn't strictly required given you cannot use "strict-local" data locality. However, if it is possible, it seems we can safely upgrade with no downtime

brainy-kilobyte-33711

05/02/2025, 9:08 AM

by removing the harvester webhook validator

rhythmic-article-81903

05/02/2025, 5:29 PM

this is what I did with last version. Have not updated to 1.5 yet

magnificent-pencil-261

05/05/2025, 2:05 AM

~~https://github.com/harvester/harvester/issues/7323#issuecomment-2577351781~~ ~https://github.com/harvester/harvester/issues/7328#issuecomment-2650037560 Users may choose to skip the check if they are aware of and accept the associated risks.~

magnificent-pencil-261

05/05/2025, 2:17 AM

Since Longhorn can block node draining, Harvester performs a pre-upgrade check to detect this condition.

magnificent-pencil-261

05/05/2025, 2:18 AM

https://github.com/harvester/harvester/pull/7361#issue-2783171559 • Before

Harvester v1.4.0

, the default LH node drain policy is

block-if-contains-last-replica

, a detached single-replica volume block the node drain • Since

Harvester v1.4.0

, we changed the LH node drain policy to `allow-if-replica-is-stopped`, even this will make the replica not block node drain, but it can still have potential data integrity concern during the upgrade

magnificent-pencil-261

05/05/2025, 2:19 AM

> The live migration moved the replica to another node In live migration, the

engine

is migrated, while the

replica

remains on the original node.

👍 1

brainy-kilobyte-33711

05/05/2025, 9:54 AM

Ah right, so our setting of "Data Locality:best-effort" meant the PVC then got rebuilt on the new node but if there isn't space I assume it will get left behind. if we temporarily change the LH node drain policy to block-for-eviction-if-contains-last-replica and delete the upgrade webhook, I think we can fairly safely upgrade without downtime of our replica 1 VMs

brainy-kilobyte-33711

05/05/2025, 9:56 AM

and having to manually move VMs around or dealing with stalled drains is much more preferable to us then having downtime of our hosted apps

Open in Slack

Previous Next