Hi everyone ! We are facing the very same problem as @busy-flag-55906 on all our clusters, upgrades are frozen with one/multiple controlplane node(s) stuck in "Waiting for probes: kube-controller-manager, kube-scheduler". Here are versions of clusters having the problem : • Rancher 2.7.1, RKE2 managed cluster upgraded from v1.24.7+rke2r1 to v1.24.9+rke2r2 (1/3 controlplane affected, 1/3 in waiting for plan) • Rancher 2.6.9, RKE2 managed cluster v1.24.7+rke2r1 (3/3 controlplanes affected) • Rancher 2.7.1, RKE2 managed cluster v1.24.8+rke2r1 (1/3 controlplane affected) • We also have one cluster not having the problem : • Rancher 2.6.9, RKE2 managed clsuter v1.24.7+rke2r1 These clusters are hosted in different hosting providers, they are all based on Ubuntu 20.04 or 22.04. We already tried to : • restart

rancher-system-agent

=> no effect • restarting nodes => no effect • upgrade to a more recent version of RKE2 even if previous was not fully deployed => no effect (still one node up-to-date with probes down) • upgrade to a more recent version of Rancher (2.6.9 => 2.7.1) => no effect Clusters are healthy from k8s point of view, ETCD cluster is healthy with all members sync, scheduling and controllers are working correctly. We are not sure if it's the root cause, but we found some articles about a change in the insecure to secure port for controller and scheduler in recent versions ok k8s, may this be a problem of wrong port used in check ? Does anyone knows if it's really rancher-system-agent that is in charge of probing scheduler and controller ? How to check probes config ? Note: This only affects rke2 clusters managed by Rancher, for cluster deployed manually with RKE2 and then imported into Rancher upgrades works without any issue.

Have you opened a support case or GitHub issue? The first being preferred if you have paid support...

We have no paid support, I can open a github issue.

@tall-translator-73410 did you look at the kubelet logs on the hosts? What pods are not running

crictl ps

and rancher-system-agent logs to see if there are any errors?

Here is rancher-system-agent log filtered on errors :

# journalctl -u rancher-system-agent -n 2000 |grep -i error
Apr 10 13:11:35 <nodename> rancher-system-agent[50346]: time="2023-04-10T13:11:35Z" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (369730905 vs 369730954)"
Apr 10 13:11:35 <nodename> rancher-system-agent[50346]: time="2023-04-10T13:11:35Z" level=error msg="error syncing 'fleet-default/custom-6ab162c666c9-machine-plan': handler secret-watch: secret received was too old, requeuing"

with crictl, I can tell that both

kube-controller-manager-<node>

and

kube-scheduler-<node>

are Running In kubelet log there are several errors talink about

failed to sync secret cache: timed out waiting for the condition

or like this one :

E0410 22:34:51.109948    1396 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-controller-manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-controller-manager pod=kube-controller-manager-<node-name>_kube-system(57585a0305e4e46df816ebab263926f3)\"" pod="kube-system/kube-controller-manager-<node-name>" podUID=57585a0305e4e46df816ebab263926f3

Go check the kube-controller-manager logs next. Also since rancher-system-agent is saying secret too old, restart rancher-system-agent

There is no error in kube-controller-manager container logs

you might want to grep kubelet.log for "error"

Yes, it is empty , the only error is about a cert-manager solver that is okay to fail :

# crictl logs 74fa73b170537 2>&1 |grep -i error
I0411 06:36:59.217837       1 event.go:294] "Event occurred" object="<namespace>/cm-acme-http-solver-v2cq5" fieldPath="" kind="Endpoints" apiVersion="v1" type="Warning" reason="FailedToUpdateEndpoint" message="Failed to update endpoint <namespace>/cm-acme-http-solver-v2cq5: Operation cannot be fulfilled on endpoints \"cm-acme-http-solver-v2cq5\": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/<namespace>/cm-acme-http-solver-v2cq5, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 7eeee381-5e72-4a7e-a24e-1089b7d40156, UID in object meta: "

check rke2-server's logs

I think it's just a wrong probe config problem. both controller and sheduler are acting good in the cluster, do you know where rancher-system-agent take its config from ?

There is much more errors in rke2-server logs

You can check rancher logs next

Here are rke2-server logs that occurs more than once :

# journalctl -u rke2-server -n 4000 |grep -i error |cut -c 27- |sed -e 's/2023-[^Z]*Z/TIMEREDACTED"/' |sort |uniq -c|sort -n |grep -v "^      1"
      3 rke2[877]: time="TIMEREDACTED"" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
      4 rke2[366270]: time="TIMEREDACTED"" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
      4 rke2[367289]: time="TIMEREDACTED"" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
      4 rke2[877]: time="TIMEREDACTED"" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
      5 rke2[877]: time="TIMEREDACTED"" level=warning msg="Proxy error: write failed: io: read/write on closed pipe"
      9 rke2[387056]: time="TIMEREDACTED"" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
     11 rke2[387056]: time="TIMEREDACTED"" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
     15 rke2[912]: time="TIMEREDACTED"" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
    162 rke2[912]: time="TIMEREDACTED"" level=warning msg="Proxy error: write failed: io: read/write on closed pipe"

So in that secret there is a probe-statuses at the bottom I imagine this is where it shows kube-scheduler/kube-controller being not ready?

yes !

so in that plan json you cans ee the current cloud-init and config.yaml files being applied

I'm going to check that (just fixing a prod issue, not related to this 🙂 )

Usually that shows up in the logs we checked before. hrm and you said only 100 days, i think the default is 365 on certs

I check it right now, (I was cross-posting results in the issue)

you guess is as good as mine 😄

(First of all, thanks a lot for your help, even if it's not the real root cause, I d'ont want to wait for the resolution to thank you 🙂)

You're welcome. Hopefully Brad can give you an quick answer, Might throw that question in general on how to fix the certs to see if anyone else has experience

Yes, as fleet seems to be stuck, I'm not sure that clicking on "rotate certificate" will actually do the job ... but maybe, let's wait for Rancher's team feedback

If you find out 😄 let me know so i can put in a maintenance plan for our clusters hahaha

haha 😄

i just check ours and they have rolled once already on our oldest cluster. Maybe during RKE2 upgrades it does it? not sure the last time you upgraded

maybe only during major upgrades ?

1.22 => 1.23 (Minor) maybe

yeah ... my bad ... when I say "major" in k8s it's minor in fact 😄

yep, k8s (1) will never change hahaha

I quite consider that "1.24" is the major number 😄 and ".11" the minor ^^

.11 is the patch

yeah ^^

haha ! excellent 😄

This sounds like an issue with the plan controller on the rancher side that causes it to constantly thrash the plan under certain conditions, which in turn causes rancher-system-agent to constantly restart rke2. This should be fixed in 2.7.2 which will release quite soon.

That sounds great ! I saw that rc10 just get out earlier, do you have any hint about release date ?

soon is the most I can say.

Ok thanks !

it’s out

Hi @creamy-pencil-82913! thanks 🙂

I upgraded to 2.7.2 today, and thought that caused the same issue... just so happens that the cluster I'm seeing this issue on is 365 days old today! 😛

Unlucky ! But it does confirm that the problem is due to cert expiration

Untitled

Your feedback is important @busy-flag-55906 🙂 It confirms that our problem is due to this cert expiration !

yes, i tried this as well with no luck

I've just tried this too, but still no good 😞

Those certs are created by the controllers themselves, they’re not managed by either rancher or rke2. I’ll have to see how they can be renewed.

Thanks @creamy-pencil-82913! It would be awesome to unlock this situation 🙂

yeah, you can’t delete the shadow pods. You would need to delete the scheduler and controller-manager manifests from /var/lib/rancher/rke2/agent/pod-manifests/ and then restart rke2-server

I mean you could probably even

pskill kube-controller-manager

and

pskill kube-scheduler

maybe?

or use crictl to delete them

I'm going to file an issue in rancher/rancher around this so we can fix this properly... this is no bueno

Deleting cert manually + force container restart with crictl is working !

You might consider changing the title of your issue now that the problem is better understood. An alternative might be: "Improve Rancher management of RKE2 clusters to handle expired kube-controller-manager and kube-scheduler certs".

If you have the problem, would you more likely find this issue with the previous title or the one you suggest ?

Great work @tall-translator-73410 glad it all go figured out!

Glad you helped 😁 👍

Thanks everyone! 😄 This sorted out the problem quickly with a couple of our clusters! 🥳

thanks, i was able to fix the issue as well on the cluster

I noticed that a standard RKE2 cluster (non downstream provisioned) appears to have different certificate locations that the downstream clusters. Anyone know if the fix steps works on that as well?

@tall-translator-73410 I find that I try to add as many relevant logs in the body of the issue, as it gets indexed by Google and people will tend to find issues that way. Thank you for filing that issue! I'll get it labeled/assigned/scheduled to fix. Currently I'm actually thinking we might try to implement a fix for this in a manner that allows an RKE2/K3s upgrade to fix it.

@tall-translator-73410 The current title for the issue can have many root causes including some that were indeed already fixed in 2.7.2. Just wanted to make sure the issue gets visibility within Rancher folks like @wide-receptionist-90874 and @creamy-pencil-82913 which already seems to be the case. Thanks for your troubleshooting efforts and sharing those with the community!

@jolly-processor-88759 yes, the action of deleting the certificate files + kicking the component should cause a new cert to be generated.

@wide-receptionist-90874 Any idea why the

/var/lib/rancher/rke2/server/tls/

kube-controller-manager

and

kube-scheduler

folders do not exists on a non-downstream provisioned RKE2 cluster? I can see it on all of our down stream provisioned systems but not our MCM cluster.

They're only present on provisioned clusters, for health checking by the rancher agent. The rancher agent configures them, they're not present by default on rke2.