https://rancher.com/ logo
Title
l

lively-zebra-61132

01/12/2023, 2:52 PM
Hi, I am having a problem with upgrading my harvester cluster. Everything went smoothly until the first node was in the
post-draining
status. At some point during that process a vm was spawned which got stuck in Containercreating because of a problem with the longhorn volume it was trying to mount. I deleted the vm launcher, the pvc and the post-drain job (because it was making it impossible to delete the vm + pvc). Now I want to restart the upgrade but when trying
kubectl delete <http://upgrade.harvesterhci.io/hvst-upgrade-gxc4t|upgrade.harvesterhci.io/hvst-upgrade-gxc4t> -n harvester-system
it fails with Error from server (BadRequest): admission webhook
"<http://validator.harvesterhci.io|validator.harvesterhci.io>" denied the request: cluster fleet-local/local status is provisioning, please wait for it to be provisioned
is there anything I can do to fix this?
q

quaint-alarm-7893

01/12/2023, 3:38 PM
hi @lively-zebra-61132 try rebooting the stuck node. i've read several issues resolved by rebooting the stuck node, and it worked for me on a different issue as well. when i rebooted it though, there didnt seem to be much indication that the node was progressing until it finished
l

lively-zebra-61132

01/12/2023, 3:42 PM
That did not help unfortunately. I think I finally broke it completely and will have to do a fresh install
q

quaint-alarm-7893

01/12/2023, 3:43 PM
bummer 😞 sorry, i'm only meh so far, still figuring out a lot and learning things myself.
l

limited-breakfast-50094

01/13/2023, 11:34 PM
Hi @lively-zebra-61132 have you already started doing a fresh install? If not I'd like to help, I work on the Harvester team and am currently acting as community coordinator
This issue looks related, does the discussion there look relevant to your problem?
l

lively-zebra-61132

01/16/2023, 3:24 PM
Hi @limited-breakfast-50094 thank you for reaching out. Yes we have already done a fresh install. In the end I should not have deleted the
post-drain
job and tried to fix the issue with the VM instead. In order to prevent this issue in the future harvester could alert users if longhorn is unable to schedule sufficiently large volumes before an upgrade or a warning about having enough space in longhorn disks could be included in the documentation.
l

limited-breakfast-50094

01/17/2023, 6:40 PM
Sounds good, can you file a feature request in github.com/harvester/harvester/issues/ ?