This message was deleted.

Depending on how the node was originally set up, you may not be able to. Ref: https://docs.harvesterhci.io/v1.4/host/#role-management • Worker: Restricts a node to being a worker node (never promoted to management node) in a specific cluster. I am interested to see if anyone has found a method for doing this (short of removing the worker node and rebuilding it with the "Manager" role).

The Problem is, that a third control-plane node was originally deployed but was deleted when trying to start an upgrade and now the automativ promotion job was running but was not successful in promoting the node... Now the upgrade startet but it is stuck because there is only 2 control-plane nodes but it needs to schedule three pods for Rancher, Harvester and Harvester-Webhook I am at a loss

Best of luck, and please share findings/lessons learned. I typically deploy Harvester as a 3-node cluster (all with manager roles) in a lab environment, but have been considering moving to 5 nodes with the manager role for stability. The underlying etcd being potentially corrupt or left in a RO state if I lose a node can make for sleepless nights in production. Backing up etcd should help.

Thanks. I am hoping Rancherlabs can help here... I really like the whole Rancher, Harvester, RKE2 Ecosystem but sometimes one feels really helpless when the stuff is not working automagically as it usually does

@miniature-lock-53926 why upgrade when you have a split control plane? The team could help if the promotion failed. You should be able to cancel the upgrade - at what point is it stuck?

I really did not want to upgrade but it started by accident while I was preparing the github issue and wanted to take a screenshot of the original error. We were already discussing aborting the upgrade but wanted to wait for a response from Rancherlabs.

There will certainly be a need to have a proper cluster. Where is the upgrade stuck on?

It is stuck trying to schedule a third instance of harvester, harvester-webhook and rancher

Ok, I see. We’d have to see what others say. It looks like everything upgraded already. It’s just the actual nodes at the moment. Have you tried to join it back in?

It was the host3 ... we accidentially deleted it while deleting a machine/x-cluster CR that was stuck in reconciling state and afterwards we deleted it cleanly from the cluster... it is still there and has harvester 1.3.1 installed but will not rejoin the cluster

I’d wipe it out and try join it. But with the version you are upgrading to.

and we are still not sure if it is safe to abort the upgrade

Yes, it should be. The services were upgraded. It’s just the nodes. But perhaps wait for someone to chime in.

You just did chime in... but you probably mean some on the github issue, right?

Why is

haa-devops-harvester02-node02

and

haa-devops-harvester02-node05

cordoned? Was this part of the upgrade or was this done manually?

No this is done via the update. If I uncordon it manually it deploys the pending pods for a minute and then cordons the node again and reschedules the pods

Could you generate the SB?

We already have. I sent Rancherlabs the download link to harvester-support-bundle@suse.com today because I could not send the SB via email directly

You can upload SB here. I thought that would be more convenient.

I picked the wrong file, it is already uploading 🙂

No worries, the upgrade log also helps. But SB could simulate your current environment.

Ah, I never knew you had like a lab-simulator where you can drop the SB into... nice

Just in my local environment. 😆 https://github.com/rancher/support-bundle-kit could help simulate.

That is quite useful. BTW are there any secrets embedded in the SB that should be changed afterwards or is it not necessary?

No, we did not collect the secrets

Here is the current state

hmm, did you want to upgrade or drop the upgrade?

If we could drop it safely I think we would prefer it

The instance-manager pod (from Longhorn) was protected by PDB until all replicas on the draining node were moved to other nodes.

Ah ok, I did not see that. What would you recommend? Roll back or go forward?

Most of the components were upgraded. So, I am not really sure whether dropping the upgrade is good or not.

Well then we would need to shake something lose to get the upgrade rolling again, I guess

Hmm, I don’t think this fleet complaint would affect the upgrade at this moment.

yeah that is why I originally thought that it would be the first priority to get one of the 2 worker nodes promoted to control-plane but I dont know how I would do that... except for retriggering the failed promotion job for host5

I am checking the code. IIUC, the promote controller should not be blocked during the upgrade so I am checking.

The running promote pods had specific errors but we thought maybe those might be "normal" and decided to let it run a litte longer but then those pods were deleted and only the failed job remained, which I did not notice until yesterday when it was to late

The running promote pods had specific errors

Did you mean you have logs of promote pod?

thats why I was thinking about just rerunning the promote job, but that is even better

it’s weird, the corresponding label was not be added

yeah, our conclusion was, that because the x-cluster CRs and the RKEControlPlane local was not finished reconciling that this blocked the promotion from completing

From the log

Waiting for promotion...

That means everything should be settled. We just wait for the status change.

yeah well after the promotion pods were no longer running, I just assumed that the promotion was then successful but host 5 didnt have the right node-roles.... and that is when I started the writeup of the Issue and while trying to get a screenshot of the original error, this upgrade was triggered accidentally

hmm, if the promotion is successful, the promote job should be completed

but it did not. Do you think that the Reconciling Cluster CRs were at fault and should I try rerunning the promotion job?

I am checking the label, seems some labels were correct

@miniature-lock-53926 Reading your comment of, "TBH I don't even understand how we ended up in this weird edgecase state." reminded me that I ran into a similar state that led to my discovering that one of my three Dell R740XD2's had a different processor generation than the other two. This led to the upgrade VM being unable to migrate to the server with the different processor and resulted in a hung upgrade state. Ref: https://github.com/harvester/harvester/issues/7096 Not saying that this is your issue, but something to watch out for (heterogenous hardware).

I need some time to discuss this situation (means promotion failure). Will update here tomorrow.

Perfect I also am finished with work today, I will be here again tomorrow morning CET

especially because after accidentally triggering the deletion of one of the control-plane nodes

Deleting a single control-plane node in a cluster with three control planes is a really awkward edge case. etcd requires the majority of control-plane nodes to be in good working order to maintain quorum and accept writes, otherwise it will fall back into a read-only mode to preserve data consistency and avoid a split-brain scenario. The minimum number of nodes required for etcd can not be changed willy-nilly, usually it can only be done by backing up the data and restoring into a completely fresh etcd instance. With one out of three control-plane nodes gone, this puts the etcd in a weird spot, where it requires both other nodes to still be there to maintain the quorum. I found an older Rancher issue about promoting RKE2 workers to master nodes, but none of what's discussed there seems to be out of whack in this case. https://github.com/rancher/rancher/issues/36480#issuecomment-1039253499 I'll need some more time to look for a smoking gun and check back with Vincente before I can advise about what to do next

@miniature-lock-53926 so from the support bundle, i can see that

host05

was scheduled for node promotion, but what i don't get is why (from the kube-controller-manager logs) the promotion job was re-enqueued multiple times (within a 5-minute timeframe) even after a promotion job pod was started

I am actually not 100% sure what the promotion pod did the first time, but I think I remember it had really specific errors, that convinced us that the promotion was stuck and thats why we activated and deactivated the the maintenance mode on host 5 in an attempt retrigger the promotion pod after it got stuck or (now come to think of it) maybe was just taking longer than we expecteded while throwing unrelated/unimportant error or warnings and we just thought it was stuck. At that point we did not understand what controls the promotion and were not aware that there was a promotion job in the first place. In any case that again restarted the promotion process and new promotion pods were being created and afterwards we decided to wait longer this time and after a while no promotion pod was running anymore and thats why I originally thought the promotion went through this time until I found the failed promotion job after the upgrade was started.

I still don't understand why

host05

is cordoned. It doesn't seem like it's being drained, but I think this may pose a problem with scheduling the required pods to complete the promotion, because while they may tolerate taints with

NoExecute

effect, they may not tolerate a taint with

NoSchedule

effect:

│ ~/Downloads/stuck_upgrade/supportbundle_207a51d7-61ff-4f36-8785-38454b6ce253_2025-01-09T15-25-27Z │ 130 ► yq '.items[] | {"name": .metadata.name, "taints": .spec.taints}' yamls/cluster/v1/nodes.yaml 
name: haa-devops-harvester02-host01
taints: null
name: haa-devops-harvester02-host02
taints:
  - effect: NoSchedule
    key: <http://kubevirt.io/drain|kubevirt.io/drain>
    value: draining
  - effect: NoSchedule
    key: <http://node.kubernetes.io/unschedulable|node.kubernetes.io/unschedulable>
    timeAdded: "2025-01-08T21:14:36Z"
name: haa-devops-harvester02-host04
taints: null
name: haa-devops-harvester02-host05
taints:
  - effect: NoSchedule
    key: <http://node.kubernetes.io/unschedulable|node.kubernetes.io/unschedulable>
    timeAdded: "2025-01-08T21:10:18Z"

The promotion process specifically looks for a taint with

NoSchedule

effect, but with a different key:

# make sure we should not have any related label/taint on the node
      if [[ $ETCD_ONLY == false ]]; then
        found=$($KUBECTL get node $HOSTNAME -o yaml | $YQ '.spec.taints[] | select (.effect == "NoSchedule" and .key == "<http://node-role.kubernetes.io/etcd=true|node-role.kubernetes.io/etcd=true>") | .effect')
        if [[ -n $found ]]
        then
          $KUBECTL taint nodes $HOSTNAME <http://node-role.kubernetes.io/etcd=true:NoExecute-|node-role.kubernetes.io/etcd=true:NoExecute->
        fi
        $KUBECTL label --overwrite nodes $HOSTNAME <http://node-role.harvesterhci.io/witness-|node-role.harvesterhci.io/witness->
      fi

While the

etcd

pods have a toleration for the

NoExecute

taint, they don't have one for the

NoSchedule

taint, which is why I think they won't start on a cordoned node and as a result, a cordoned node won't successfully get promoted. There is also a Harvester and a Harvester webhook pod which can't be scheduled ever since

host05

was cordoned:

...
  status:
    conditions:
    - lastProbeTime: "null"
      lastTransitionTime: "2025-01-08T21:10:18Z"
      message: '0/4 nodes are available: 2 node(s) didn''t match pod anti-affinity
        rules, 2 node(s) were unschedulable. preemption: 0/4 nodes are available:
        2 No preemption victims found for incoming pod, 2 Preemption is not helpful
        for scheduling..'
      reason: Unschedulable
      status: "False"
      type: PodScheduled

Yeah right. But I thought that canceling the Upgrade should uncordon Host5 again and than the promotion could/should finish or am I missing something here? But the problem is also, that the 3rd control-plane node Host3 was removed from the Cluster on Tuesday 07.01 at around 4 PM (CET) but the promotion was not sucessfully finished on Wednesday 08.01 at around the same time. That was when we saw the promotion pod and thought it was stuck and apparently "retriggered" it turning Maintenance mode On and Off again on Host5

I don't think we're going to see much. The promotion job runs a script, which you'll find in the configmap

harvester-system/harvester-helpers

Yeah I have found that later and was also tempted to just run the script direktly on the node or to rerun the failed job again, but have not done either yet

Re-running the script won't do much, since it won't solve the problem of why the finishing condition is never reached. My suspicion is that the taint on

host05

prevents the pods that make up a Harvester master node from being scheduled. As a result, the node never finishes the promotion. That's why I want to know why the taint was put there in the first place, because then I can maybe tell if it's safe to remove, which would perhaps unblock the promotion. Right now none of my colleagues are online, but later Ivan will be. I'll ask him what he thinks of this.

Ok. If you need any more information or want me to get some log or want to debug via a Screen Session just let me know. I will be available when your colleagues are on again. One last question, would you agree with the emerging consensus, that at least the upgrade could be aborted at this current state without making things much worse? And thank you so much too everyone that has helped so far. I am really grateful for all the detailed and knowledgeable Feedback from everyone. I really appreciate the effort even though we dug ourselves into quite a mess 🙂

from what we can see, it looks like most of the components were already upgraded. i feel like the least invasive thing to do now is to add new 1.3.2 control plane nodes to the cluster, and let k8s finish scheduling those pending harvester pods.

Hi again, thanks for chiming in. But now I have conflicting advice, because @bland-farmer-13503 and @happy-cat-90847 adviced against adding the third control-plane while the upgrade is still running, or did you also mean that I should abort the upgrade first?

seems to me that most aspects of the cluster is already upgraded and the cluster is still reachable. the only thing that is missing is the pending harvester pods.

I think so too. I am just being hesitant because some of your colleagues think that continuing while the upgrade is still running is a bad idea, and I am willing to do either option but I do not really have time pressure to get it back up ASAP and thats why I am willing to wait until everybody is on the same page or until the risks for each approach are clearer to me

last you posted, your upgrade is already at phase 4, with 'Upgrading System Service' already completed, right?

yes

this is the current state

Hi staedter, we've been chatting internally about this a bit and the consensus seems to be that in order to get anything moving forward, it's best to get back to three control-plane nodes. However it's a bit unclear how you can get there, because: 1. It's unclear what will happen with the promote job when a third control-plane node is joined 2. It's unclear if the promotion, or a join will succeed, since the

<http://rkecontrolplane.rke.cattle.io|rkecontrolplane.rke.cattle.io>

resource has been deleted 3. If joining another control-plane node, it's unclear (to me) if it should be of version v1.3.1 or v1.3.2 for best chances of success. I had asked some colleagues from the Rancher team about the

<http://rkecontrolplane.rke.cattle.io|rkecontrolplane.rke.cattle.io>

CRD late on Friday, but I haven't received an answer yet.

Hi Moritz, thank you for the update. I really appreciate all the timely feedback and your effort helping us. As I said we have the "luxury" to wait a bit more and as long as you are discussing it internally I can and will resist the urge to make it even worse ;)

Is it not deleted? I was under the impression that it is. Sorry, this must have been a misunderstanding. From https://github.com/harvester/harvester/issues/7331 the reproduction steps:

3. Check the states of some Custom Ressources like Machines or RKEControlPlanes`and see that they are stuck in a`Provisioning`or`Reconciling` state .

4. Delete the stuck CRs, which triggers the deletion of one of the control-plane nodes.

To me this implied that the

rkecontrolplane

resource was deleted. But you're right, I should have double checked with the support bundle, it's indeed not deleted.

Oh sorry I did not realize that I did not specify correctly which CRs I deleted. I will make it clearer in the original Issue ... I was talking about the stuck machines.x-cluster Ressources... not even I would have been foolish enough to delete the whole rkecontrolplane of the Harvester Cluster xD That happended to me once or twice in a downstream cluster, and I learned the hard way that there is no coming back from this (at least I have not found any way)

Yes, it does indeed. It lowers my worries that joining another control-plane node may fail quite a bit. At the moment, I'm trying to go through that exact scenario in my virtual dev environment. I have set up a 3 control-plane, 2 worker Harvester v1.3.1 cluster and then deleted the

<http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io>

object belonging to one of the nodes. Then I'm trying to join back a new node in place of the old one. I'm ignoring the upgrade for now to make my test easier to setup. For a worker node joining a deleted machine back has worked flawlessly, but for a control-plane node I haven't seen it work well yet. But my first attempt wasn't clean as my workstation ran out of memory, so I'll try again. One thing I already noticed is that if one of the two remaining control-plane nodes experiences any kind of trouble the cluster pretty much immediately becomes inoperable, since the etcd store loses quorum.

Yeah that would make sense... and that is why it is a good thing the the upgrade stalled where it has and has not yet drained and rebooted one of the control-plane nodes because then the etcd would have been inoperable, right?

Yes. And if the etcd becomes inoperable the situation will be a lot less enjoyable than what we have right now.

Ok, got it. I will you let you test some more. I guess then the main question now would be what has better chances to succeed: add host3 with 1.3.1 or 1.3.2

Hi, I tried reproducing your situation in my dev environment, but I couldn't get the exact same failure scenario, so I tried some simplified scenarios to see what should work and what certainly won't. Here are some insights: 1. You can remove nodes from a Harvester cluster by simply deleting the

<http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io>

resource. Once the cluster has finished reconciling (and the

node

resource is also gone), you can join back a new node under the same name by re-installing on fresh hardware and using the node-join-token. This works both for worker nodes as well as control-plane nodes (tested on v1.3.1 when no upgrade is running though). 2. I also tried deleting the

<http://machine.cluster.x-k8s.io|machine.cluster.x-k8s.io>

resource of a control-plane node during a running upgrade, but I likely did this during a different phase of the upgrade than you. I was able to join the node back using the previously described method of doing a clean install, using the node-join-token to join the node. Once my cluster had 3 control-plane nodes again, the upgrade erred out, but the cluster seemed healthy and I was able to re-start the upgrade. Unfortunately the second attempt at the upgrade didn't succeed (the API server kept crashlooping for ~2h before I pulled the plug on this experiment). During the first upgrade attempt, one of the worker nodes entered a failed state, but I was able to reboot it to get it back to a healthy state. I'm pretty sure this was a resource starvation problem. Ivan and Alejandro suggested when joining the third control-plane, to go directly with v1.3.2. You should be able to fetch the node join token out of

/etc/rancher/rancherd/config.yaml

on one of the existing control-plane nodes. I wish you good luck, since I can't give you a guarantee that this will resolve the cluster's problems.

Thank you. We are now getting ready to try this and get some of the more important test data backed up and than will try to rejoin host 3 with version 1.3.2 I will keep you updated here and in the Github Issue if encounter more problems or if the the plan worked out.

yip

is the cloud-init clone that is used by Elemental, which is the base OS installer used in Harvester: https://github.com/rancher/yip But I don't think that is really the root problem here. I'm also assuming that you're using network settings that you already know are good. Are you using a remote-mounted ISO image?

We have just retried it with a remote-mounted image an got the same error. We had problems with the remote media in the past, that is why we switched to USB sticks

Remote mounted media can show this kind of issue, if it times out.

we are currently trying another version of the installation medium remote mounted just to see if the issue persists there...

And no, the installation will not necessarily wipe the partition table. It's optional, and IIRC only some of the more recent installers support it at all.

I actually would rather not wipe the whole disk becaus we used 2.4 TB of this disk as the default longhorn disk... most our data is on extra disk that are served by a custom storage-class but there might still be data on that partition but we will try this too if it gets the installation rolling

If the issue persists with USB sticks, I'd first try to find out what exactly the error is. You can log into the running installer image: https://docs.harvesterhci.io/v1.3/troubleshooting/index/#logging-into-the-harvester-installer-a-live-os Then check the usual places for logs etc. If all else fails, you generate a tarball with debug info with:

supportconfig -k -c

in the installer.

I just doublechecked unfortunately we included the default disks into our storage-class. We still can wipe the disk... then the longhorn data has to rebuild the missing replicas, cant be helped...

Weird. Is

nvme9n1

an NVMe-oF device or something like that? The Harvester installer can be quite tricky if there are things like that floating around, since it mounts partitions by label. Usually it's not a problem, but in some cases the EFI may expose partitions whose labels match.

Yes we think that might have been the problem now and also might have explained other weird phenomena in the past

Crossing my fingers for you 🤞

ok, so the upgrade with 1.3.2 on host3 with a remote-mounted medium was successfully and we are now waiting here for like 5 minutes already

Do you see the node in the cluster web UI, or is it not there either?

ah ok... we should have used the fqdn i guess... but it was the same value we had set before in the original harvester.config

I mean, does the node show up when you do a

kubectl get nodes

?

we will try to change the server url in the harvester.config and 90_custom.yaml to match the fqdn

Looks like going with the FQDN is the way.

rebooting it now

I wouldn't be worried about seeing those errors a few times. They basically tell you that something that acts like a K8s client failed to watch for a resource, which can happen for a variety of common reasons and the client should know to to handle it and retry. You should see the node joining the cluster

How long should we wait before we see anything inside the harvester cluster? The management url is still "Not ready" as before and the watch on

k get node

and

k get pod -A --field-selector status.phase!=Running

has not shown any change at all and the kubelet throws the same errors every other minute

Maybe like five minutes? You did choose to "join an existing Harvester cluster" in the installer, right?

You used version 1.3.1 for your tests, right? Maybe that was right after all?

Indeed, I only used v1.3.1

Probably because the k8s components on the remaining control-plane nodes were not upgrade and are still running in older versions

Yep.

v1.27.13

is the RKE2 version that powers Harvester v1.3.1

Ok we are trying it with 1.3.1 now... because of remote situation problem we wanted to try the netinstall version but had ah dev/kvm error that we have not encountered before and are now trying again with the regular iso but then then installation alone takes almost an hour

Yeah, unfortunately the installation is everything but quick.

Ok, I guess now we have the same problem as before, that there is yet again another installation medium (the faulty usb stick that does not boot) that is picked up by the installation routine, so we are not sure which version is actually used during installation, and we assume that the

dev/kvm is missing

error is just a symptom of that, like with the

yip version -g

error before- so we just have to wait for some hands on-site to update the usb stick to 1.3.1 and flash it with rufus and we will continue here tomorrow Not great; not terrible ... atleast we have not made things worse yet

Good morning. What does

journalctl -u rke2-server.service

show? Is that unit even running?

no it is not even running

Why would the management URL not be healthy? The API service should be redundant between the control-plane nodes and the ingress should switch to one of the remaining nodes as a backend. Can't you reach the Kubernetes API?

We can reach the API but the management api is unhealthy...

But this is on

host03

. What about the other hosts? There the management API should be healthy.

ah ok sorry for the misunderstanding... I have to look in the IPMI for another server...

You should be able to

curl

it from

host03

yeah but how do i authenticate from it?

You don't need to. This shows us that

host03

is able to connect to the API and that the API is there. So something else failed when re-installing

host03

causing it to not be able to join the cluster.

Ok but from one of the healthy nodes Servers Dashboard looks like this. What exactly determines if the management-url is ready from a nodes perspective?

Yeah... I was wondering how much pain you're willing to go through with this cluster, especially since I though this was an evaluation cluster, not for production workloads. The dashboard on the console literally just does a

curl

on the management URL and checks if the return code of that process indicates success or not.

Well it was almost production-ready... the last thing before our new flag-ship application was supposed to run there was the those tests, that thankfully are still able to run and are producing really promising results... and the upgrade to 1.4.0 xD

I just skimmed the code real quick. The

curl

is the last step in a series of checks. These checks are all looking for objects in the K8s API. These checks all work together to make sure that the status is displayed correctly whether you're looking at the dashboard on the first node of a cluster or whether you're looking at the dashboard of a worker node, etc. Not sure why there needs to be another

curl

request at the end right now. But if any of these checks fail for any reason, the dashboard will not show the cluster as "Ready". So in a way it's showing the correct info.

host03

isn't ready because it's not properly joined in the cluster, but the other are all showing ready because the cluster is essentially still operating.

yes that makes sense... but how would a new node check on k8s ressources via the api when it cannot authenticate. this is still a mistery to me

It just won't be ready until it has finished joining the cluster. Part of the joining process is to configure these authentication credentials.

ah ok got it