This message was deleted.

can you provide a support bundle?

Bundle attached

seems Rancher is waiting for kubelet on first node

- lastUpdateTime: "2024-06-16T21:43:45Z"
    message: 'configuring bootstrap node(s) custom-9cb22ccf7984: waiting for kubelet
      to update'
    reason: Waiting
    status: Unknown
    type: Provisioned

The node is

harvester1

but kubelet seems to work well and up there.

yes that I noticed

if you don't have any workload on harvester1, maybe try restarting rke2-server on it and see if it helps?

Restarting did not help

Thanks, we'll need time to check. Please don't delete anything 🙏

any updates?

Hi, Rancher team will check. They requested some logs, which I can only share in my time zone. From my observation, the rancher is waiting kubelet version to become v1.27.13. but the machine object still shows it's 1.27.10.

- lastUpdateTime: "2024-06-16T21:43:45Z"
    message: 'configuring bootstrap node(s) custom-9cb22ccf7984: waiting for kubelet
      to update'
    reason: Waiting
    status: Unknown
    type: Provisioned

# machine custom-9cb22ccf7984, which is node harvester1
  nodeInfo:
    architecture: amd64
    bootID: 45075668-4501-4b4d-bd39-6f133b567b02
    containerRuntimeVersion: <containerd://1.7.11-k3s2>
    kernelVersion: 5.14.21-150400.24.108-default
    kubeProxyVersion: v1.27.10+rke2r1
    kubeletVersion: v1.27.10+rke2r1  <-----

# rkeConfig in cluster object
spec:
  kubernetesVersion: v1.27.13+rke2r1 <----
  localClusterAuthEndpoint: {}

The weird part is node harvester1 is already upgraded.

$ kubectl get nodes
NAME         STATUS   ROLES                       AGE   VERSION
harvester1   Ready    control-plane,etcd,master   49d   v1.27.13+rke2r1     <- correct version
harvester2   Ready    control-plane,etcd,master   49d   v1.27.10+rke2r1
harvester3   Ready    <none>                      48d   v1.27.10+rke2r1
harvester4   Ready    <none>                      48d   v1.27.10+rke2r1
harvester5   Ready    control-plane,etcd,master   49d   v1.27.10+rke2r1
harvester6   Ready    <none>                      48d   v1.27.10+rke2r1
harvester7   Ready    <none>                      48d   v1.27.10+rke2r1

any updates on this? This is causing quite a lot headache for us now...

Hi @rhythmic-article-81903, could you add

cattle-provisioning-capi-system

to support-bundle-namespaces setting and generate support bundle again? The machine node info is different from node status. I suspect there may be some error in capi-controller-manager deployment. Thank you. https://github.com/kubernetes-sigs/cluster-api/blob/00dbf7b9f6322d7ebd06ae2efa703b[…]d37d/internal/controllers/machine/machine_controller_noderef.go

Attached

Yeah, there are some error message in it. Hope it works.

now it is progressing

@rhythmic-article-81903 Sorry to bother you again, we are still checking the root cause of this issue. Was the cluster a new v1.3.0 installation? Means it's not upgraded from any previous v1.2 .xversions. Thanks.

it was new 1.3