Hi all ; i am trying to add new worker node in my existing rancher cluster but getting below error . i really dont have any clue .connectivity is ok openssl s_client -connect rancher.lab:443 (no issue with ssl connectiity) Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="[Prober] (kubelet) running probe" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="[Prober] (kubelet) retrieving existing probe status from map if existing" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="Probe timeout duration: 5 seconds" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="Probe output was Get \"http://127.0.0.1:10248/healthz\": dial tcp 127.0.0.1:10248: connect: connection refused" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="Setting success threshold to 1" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="Setting failure threshold to 2" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="Probe failed" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="[Prober] (kubelet) writing probe status to map" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="[K8s] Enqueueing after 5.000000 seconds" Mar 23 04:39:57 pol1.lab rancher-system-agent[11037]: time="2023-03-23T04:39:57Z" level=debug msg="[K8s] secret data/string-data did not change, not updating secret"

adding worker nodes requires a lot more ports than just 443, did you check the port permission list for 6443 and 9345?

i have not restricted any ports in my server. even no firewall

what server os is it?

SUSE linux

Is flannel injecting into iptables? Sudo iptables -L

No it is not injecting. Here is the output from my worker ..iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination

I know there is a known issue w flannel and ubuntu 22.04, maybe its related? Is the new node iptables version different than the others?

no .. its all SUSE rancher OS installed blades on same chassis

Either way whats the iptables version on a good vs bad node, i just want to verify its not that

iptables --version iptables v1.8.7 (legacy)

I still think its some kind of lack of permissions or something with flannel, but 1000% please let me know if you find a fix, ive been fighting this same type of issue 😅

ok sure