Good afternoon folks, I have been seeing an issue ...
# rke2t
tall-raincoat-70627
05/15/2025, 10:47 PMGood afternoon folks, I have been seeing an issue where on install it appears that etcd is failing to start, and the bootstrapping just loops around attemtping to connnect to etcd with no success. The image is on the server according to crictl but no pods are running, and the rke2 logs are not showing anything seemingly useful, even with hte debug flag, is there anything I can do to increase the errors being returned from the initial attemps to start etcd?
c
creamy-pencil-82913
05/15/2025, 10:52 PMcheck the kubelet and containerd logs
creamy-pencil-82913
05/15/2025, 10:52 PMwhat kind of resources (cpu/memory) do you have on the node?
t
tall-raincoat-70627
05/15/2025, 10:53 PM2CPUS 32G of RAM
c
creamy-pencil-82913
05/15/2025, 10:53 PMyou might also check the etcd logs under /var/log/pods
creamy-pencil-82913
05/15/2025, 10:53 PM2 cores or 2 sockets?
creamy-pencil-82913
05/15/2025, 10:53 PMthat’s a lot of memory for just 2 cores. You need at least 4 cores for a server.
t
tall-raincoat-70627
05/15/2025, 10:54 PMok, i can bump it up
c
creamy-pencil-82913
05/15/2025, 10:55 PMthat’s probably not it, but thats definitely low for a server.
t
tall-raincoat-70627
05/15/2025, 10:55 PMIt doesnt appear to be getting far enoiugh to create /var/log/pods
c
creamy-pencil-82913
05/15/2025, 10:56 PMdefinitely check containerd and kubelet logs then
t
tall-raincoat-70627
05/15/2025, 11:00 PMTHe only non info message is:
Copy code
time="2025-05-15T22:45:15.024459505Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"c
creamy-pencil-82913
05/15/2025, 11:00 PMcni doesn’t get installed until later
creamy-pencil-82913
05/15/2025, 11:00 PMkubelet and containerd are both running?
t
tall-raincoat-70627
05/15/2025, 11:01 PMkublet exits when it cant contyact etcd
c
creamy-pencil-82913
05/15/2025, 11:01 PMthats not how that works
t
tall-raincoat-70627
05/15/2025, 11:01 PMand gets restarted and loops
c
creamy-pencil-82913
05/15/2025, 11:01 PMkubelet does not talk to etcd, the apiserver does
t
tall-raincoat-70627
05/15/2025, 11:01 PMok, let me try again
c
creamy-pencil-82913
05/15/2025, 11:01 PMwhat is the exact error the kubelet is exiting with
creamy-pencil-82913
05/15/2025, 11:01 PMkubelet crashlooping would definitely prevent pods from getting started
t
tall-raincoat-70627
05/15/2025, 11:03 PM Copy code
May 15 23:02:48 na-nonprod-dynenvcp-01c.atl01.xx.com rke2[387567]: time="2025-05-15T23:02:48Z" level=info msg="Failed to test etcd connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""
May 15 23:02:48 na-nonprod-dynenvcp-01c.atl01.xx.com rke2[387567]: time="2025-05-15T23:02:48Z" level=error msg="Kubelet exited: exit status 1"c
creamy-pencil-82913
05/15/2025, 11:04 PMthat’s not the kubelet log. look at the kubelet log.
t
tall-raincoat-70627
05/15/2025, 11:05 PMIf i make a change to config.yaml will that get picked up between restarts, or do i need to reinstall?
c
creamy-pencil-82913
05/15/2025, 11:05 PMjust need to restart
t
tall-raincoat-70627
05/15/2025, 11:06 PMYep, my error in the config.yaml passing it an invalid flag
c
creamy-pencil-82913
05/15/2025, 11:06 PMthat’d do it
t
tall-raincoat-70627
05/15/2025, 11:06 PMthanks for the assistance
🙌 1
tall-raincoat-70627
05/15/2025, 11:08 PM"command failed" err="failed to set feature gates from initial flags-based config: unrecognized feature gate: EphemeralContainers"tall-raincoat-70627
05/15/2025, 11:08 PMwould be the culprit
7 Views