bumpy-receptionist-35510
05/08/2023, 2:56 PM[INFO ] waiting for viable init node
[INFO ] configuring bootstrap node(s) rancher-test-pool1-5fd745d759-p6wvj: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) rancher-test-pool1-5fd745d759-p6wvj: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) rancher-test-pool1-5fd745d759-p6wvj: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) rancher-test-pool1-5fd745d759-p6wvj: waiting for probes: kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) rancher-test-pool1-5fd745d759-p6wvj: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) rancher-test-pool1-5fd745d759-p6wvj: waiting for cluster agent to connect
[INFO ] non-ready bootstrap machine(s) rancher-test-pool1-5fd745d759-p6wvj and join url to be available on bootstrap node
But here is gets stuck forever. The "Explore" button has turned blue and can be clicked and I can see the node details and a few other entities in the cluster.
But it won't complete that last step and the node has the label <http://plan.upgrade.cattle.io/system-agent-upgrader=7dbfe3bc7aa1f9e217597840afda8c191bdef89403968ed04de99eec|plan.upgrade.cattle.io/system-agent-upgrader=7dbfe3bc7aa1f9e217597840afda8c191bdef89403968ed04de99eec>
and the other nodes don't do anything either.
Right now we selected the cilium network plugin but we ran into the same issue using calico. We also don't see any obvious error messages anywhere that would point us into the right direction. Any ideas or pointer what might be causing this?
Thanksagreeable-oil-87482
05/08/2023, 5:40 PMbumpy-receptionist-35510
05/08/2023, 6:02 PM# /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-fleet-system fleet-agent-74784f4cf6-27cfs 1/1 Running 0 8h
cattle-system cattle-cluster-agent-6bf7d49647-gv4f7 1/1 Running 0 8h
cattle-system rancher-webhook-74f44ffdb9-drmvg 1/1 Running 0 8h
cattle-system system-upgrade-controller-64f5b6857-hstq7 1/1 Running 0 8h
kube-system cilium-lv5fc 1/1 Running 0 9h
kube-system cilium-operator-588b7fbb5f-fjjmv 1/1 Running 0 9h
kube-system cilium-operator-588b7fbb5f-p7csw 0/1 Pending 0 9h
kube-system etcd-rancher-test-pool1-17288e44-g2r9w 1/1 Running 0 9h
kube-system helm-install-rancher-vsphere-cpi-qcjlc 0/1 Completed 0 9h
kube-system helm-install-rancher-vsphere-csi-sfhqq 0/1 Completed 0 9h
kube-system helm-install-rke2-cilium-rlwjv 0/1 Completed 0 9h
kube-system helm-install-rke2-coredns-jhnnd 0/1 Completed 0 9h
kube-system helm-install-rke2-ingress-nginx-wwzz4 0/1 Completed 0 9h
kube-system helm-install-rke2-metrics-server-tv7nt 0/1 Completed 0 9h
kube-system helm-install-rke2-snapshot-controller-crd-67hxt 0/1 Completed 0 9h
kube-system helm-install-rke2-snapshot-controller-xznk6 0/1 Completed 1 9h
kube-system helm-install-rke2-snapshot-validation-webhook-tvb49 0/1 Completed 0 9h
kube-system kube-apiserver-rancher-test-pool1-17288e44-g2r9w 1/1 Running 0 9h
kube-system kube-controller-manager-rancher-test-pool1-17288e44-g2r9w 1/1 Running 0 9h
kube-system kube-proxy-rancher-test-pool1-17288e44-g2r9w 1/1 Running 0 9h
kube-system kube-scheduler-rancher-test-pool1-17288e44-g2r9w 1/1 Running 0 9h
kube-system rancher-vsphere-cpi-cloud-controller-manager-g7l9q 1/1 Running 0 9h
kube-system rke2-coredns-rke2-coredns-6b9548f79f-pcntr 1/1 Running 0 9h
kube-system rke2-coredns-rke2-coredns-autoscaler-57647bc7cf-92hq2 1/1 Running 0 9h
kube-system rke2-ingress-nginx-controller-ttfpx 1/1 Running 0 8h
kube-system rke2-metrics-server-7d58bbc9c6-cpvcn 1/1 Running 0 8h
kube-system rke2-snapshot-controller-7b5b4f946c-fm5rg 1/1 Running 0 8h
kube-system rke2-snapshot-validation-webhook-7748dbf6ff-7lbjz 1/1 Running 0 8h
kube-system vsphere-csi-controller-674d956d9c-54w75 6/6 Running 0 9h
kube-system vsphere-csi-controller-674d956d9c-brg2w 6/6 Running 0 9h
kube-system vsphere-csi-controller-674d956d9c-m88qg 6/6 Running 0 9h
kube-system vsphere-csi-node-xfr6l 3/3 Running 2 (8h ago) 9h
root@rancher-test-pool1-17288e44-g2r9w:/var/lib/rancher/rke2/agent#
{"name":"ping","data":{}}
as well as other event structures backagreeable-oil-87482
05/09/2023, 8:33 AMkube-system cilium-operator-588b7fbb5f-p7csw 0/1 Pending 0 9h
bumpy-receptionist-35510
05/09/2023, 8:35 AMagreeable-oil-87482
05/09/2023, 8:35 AMbumpy-receptionist-35510
05/09/2023, 8:35 AMagreeable-oil-87482
05/09/2023, 8:37 AMbumpy-receptionist-35510
05/09/2023, 8:38 AMnon-ready bootstrap machine(s) rancher-test-pool1-5fd745d759-p6wvj and join url to be available on bootstrap node
maybe I can then look at why it may not happenagreeable-oil-87482
05/09/2023, 8:38 AMbumpy-receptionist-35510
05/09/2023, 8:39 AMagreeable-oil-87482
05/09/2023, 9:00 AMbumpy-receptionist-35510
05/09/2023, 9:01 AMagreeable-oil-87482
05/09/2023, 1:03 PMbumpy-receptionist-35510
05/09/2023, 1:04 PMagreeable-oil-87482
05/09/2023, 1:30 PMbumpy-receptionist-35510
05/09/2023, 1:30 PMroot@rancher-test-pool1-17288e44-g2r9w:~# /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml run tmp-shell --rm -i --tty --image nicolaka/netshoot
on the cluster node that is stuck in provisioning. Then performing the kb article procedure from within the pod:
curl -s -i -N \
--http1.1 \
-H "Connection: Upgrade" \
-H "Upgrade: websocket" \
-H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
-H "Sec-WebSocket-Version: 13" \
-H "Authorization: Bearer $TOKEN" \
-H "Host: $FQDN" \
-k https://$FQDN/v3/subscribe
HTTP/1.1 101 Switching Protocols
Date: Tue, 09 May 2023 13:34:22 GMT
Connection: upgrade
Upgrade: websocket
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Strict-Transport-Security: max-age=15724800; includeSubDomains
β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}β{"name":"ping","data":{}}
agreeable-oil-87482
05/09/2023, 6:44 PMbumpy-receptionist-35510
05/10/2023, 7:36 AMtls.source
instead of ingress.tls.source
π€¦ββοΈ Apologies for wasting your time with this. But thank you so much for trying to help. It was so wired that the UI and everything worked just fine so I didn't expect something to be wrong with the Rancher install itself.agreeable-oil-87482
05/10/2023, 7:38 AMbumpy-receptionist-35510
05/10/2023, 7:39 AMagreeable-oil-87482
05/10/2023, 7:42 AMbumpy-receptionist-35510
05/10/2023, 7:43 AMagreeable-oil-87482
05/10/2023, 8:03 AMbumpy-receptionist-35510
05/10/2023, 8:05 AM