I am upgrading my harvester cluster 1.4.1 to 1.4.2. Node 1 is stuck in "pre-drained" state, but the Kubernets version on the node has been upgraded which, the upgrade, so I would expect to be in the "post-drain" state. How can i push the upgrade process along to recognise that this state is done? I have tried to manually restart the node in the hopes that it would advance the upgrade process, but it hasn't helped.

is

dev-hvst-n02

the one that's stuck?

https://github.com/harvester/harvester/issues/9371 link back the related GH issue

Hello. Thanks you all for the replies, and your time. We were not aware of the pre-check script. Unfortunately the script made us aware of expired certificates. --- Output of check.sh /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt info: Validity Not Before: Apr 17 113400 2024 GMT Not After : Apr 17 113400 2025 GMT /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt has already expired. /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt info: Validity Not Before: Apr 17 113400 2024 GMT Not After : Apr 17 113400 2025 GMT One or more of your certificates have already expired. Typically certs should auto-renew when the RKE service is restarted (This can also happen when you reboot a node) You can also trigger a rotation of all a nodes' service certificates by running: kubectl edit clusters.provisioning.cattle.io local -n fleet-local and adding +=1 to the spec.rkeConfig.rotateCertificates.generation field or set it to 1 if it's missing. This should be done before an upgrade. If an upgrade is already in process you might follow the workaround listed in this GitHub Issue: https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311 Certificates Test: Failed --- We have tried to apply the fix mentioned in the github issue. This moved the upgrade process a bit further along. Putting node03 into cordoned mode, but the certificates on node02 and node03, are also expired. We decided to update the certificates on all 3 three nodes. As node 3 was already cordoned, we updated the certificate on node 3, And lastly on node 2. This resulted in 95% of the VM in Harvester to switch off. We were unable to turn them on again, as they turned off again, 10 seconds after being turned on. After restarting node 2 we were able to turn on VMs again, and get the clusters online. ----- Current status is, that node 3 is stuck in cordoned state, and puts itself back in that state, even if we manually uncordon it. Probably the upgrade process getting that node ready for the upgrade. However, it is not being put into maintenance mode. Is there a way to way to restart the Upgrade process, or delete the upgrade CR. We tried deleting it, but got ad error saying that the upgrade cannot be deleted because the upgrade in in progress.

Hi @brash-author-46146, sorry for late reply. I suggest you to reply in the above github issue and provide support bundle for us to do further analysis. Thanks!