https://rancher.com/ logo
s

strong-france-26978

03/23/2023, 5:04 AM
Hi all ; i am trying to add new worker node in my existing rancher cluster but getting below error . i really dont have any clue .connectivity is ok openssl s_client -connect rancher.lab:443 (no issue with ssl connectiity) Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="[Prober] (kubelet) running probe" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="[Prober] (kubelet) retrieving existing probe status from map if existing" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="Probe timeout duration: 5 seconds" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="Probe output was Get \"http://127.0.0.1:10248/healthz\": dial tcp 127.0.0.110248 connect: connection refused" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="Setting success threshold to 1" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="Setting failure threshold to 2" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="Probe failed" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="[Prober] (kubelet) writing probe status to map" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="[K8s] Enqueueing after 5.000000 seconds" Mar 23 043957 pol1.lab rancher-system-agent[11037]: time="2023-03-23T043957Z" level=debug msg="[K8s] secret data/string-data did not change, not updating secret"
p

polite-piano-74233

03/23/2023, 5:32 AM
adding worker nodes requires a lot more ports than just 443, did you check the port permission list for 6443 and 9345?
s

strong-france-26978

03/23/2023, 5:34 AM
i have not restricted any ports in my server. even no firewall
p

polite-piano-74233

03/23/2023, 5:34 AM
what server os is it?
s

strong-france-26978

03/23/2023, 5:55 AM
SUSE linux
p

polite-piano-74233

03/23/2023, 6:11 AM
Is flannel injecting into iptables? Sudo iptables -L
s

strong-france-26978

03/23/2023, 6:21 AM
No it is not injecting. Here is the output from my worker ..iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
Any issue with certificates exchange you think ?
p

polite-piano-74233

03/23/2023, 6:24 AM
I know there is a known issue w flannel and ubuntu 22.04, maybe its related? Is the new node iptables version different than the others?
s

strong-france-26978

03/23/2023, 6:25 AM
no .. its all SUSE rancher OS installed blades on same chassis
p

polite-piano-74233

03/23/2023, 6:26 AM
Either way whats the iptables version on a good vs bad node, i just want to verify its not that
s

strong-france-26978

03/23/2023, 6:44 AM
iptables --version iptables v1.8.7 (legacy)
cat /etc/os-release NAME="SLES" VERSION="15-SP3" VERSION_ID="15.3" PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3" ID="sles" ID_LIKE="suse" ANSI_COLOR="0;32" CPE_NAME="cpe/osusesles15:sp3" DOCUMENTATION_URL="https://documentation.suse.com/"
both version same in good and bad nodes
p

polite-piano-74233

03/23/2023, 7:11 AM
I still think its some kind of lack of permissions or something with flannel, but 1000% please let me know if you find a fix, ive been fighting this same type of issue 😅
s

strong-france-26978

03/23/2023, 7:12 AM
ok sure
found the issue . i have a token mismatch in 50-rancher.yaml file which resides under /etc/rancher/rke2/config.yaml.d "token": "klkwf9x7dvs89h7jqfrq4bg5vmt6mtdchwdcvf49xcfmtdxtwwlq5m"
🙌 1