https://rancher.com/ logo
Title
h

hallowed-hair-79157

06/17/2022, 12:54 AM
hello, I'm trying to set up a bare metal rke2 cluster and I am having issues adding new server nodes to my cluster. when I review the logs for rke2-server, I get the following error:
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: F0616 21:53:06.418489    7641 csi_plugin.go:305] Failed to initialize CSINode after retrying: timed out waiting for the condition
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: goroutine 796 [running]:
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.stacks(0xc000a0e101|k8s.io/klog/v2.stacks(0xc000a0e101>, 0xc0002122c0, 0x82, 0x291)
Jun 16 21:53:06 test-rke-server-1 rke2[7641]:         /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.(*loggingT).output(0x7648520|k8s.io/klog/v2.(*loggingT).output(0x7648520>, 0xc000000003, 0x0, 0x0, 0xc000586540, 0x0, 0x618e831, 0xd, 0x131, 0x0)
Jun 16 21:53:06 test-rke-server-1 rke2[7641]:         /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x1e5
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.(*loggingT).printf(0x7648520|k8s.io/klog/v2.(*loggingT).printf(0x7648520>, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x4efe5d9, 0x2f, 0xc0016f2cb0, 0x1, ...)
Jun 16 21:53:06 test-rke-server-1 rke2[7641]:         /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:753 +0x19a
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.Fatalf(...)|k8s.io/klog/v2.Fatalf(...)>
(error trace goes for hundreds of lines) has anyone run into this issue before/know what is wrong? was not able to find much from googling. I am running k8s/rke2 version 1.22.10+rke2r2 on centos 8 stream note that this only happens on the second server node, my first node is fine:
NAME                STATUS   ROLES                       AGE     VERSION
test-rke-server-0   Ready    control-plane,etcd,master   4h44m   v1.22.10+rke2r2
never mind... found the issue lol. the very bottom of the go error trace referenced a dns related go module, and that's when I realized/remembered that my nodes were pointing to an invalid dns in their ifconfig settings (I was trying to do a airgapped install). Once I removed the DNS settings entirely, everything worked as expected!