https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
c
what version of k3s are you using? what do the logs say before that regarding flannel startup?
a
Copy code
$ k3s --version
k3s version v1.32.8+k3s1 (fe896f7e)
go version go1.23.11
Copy code
Dec 01 23:40:03 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:03Z" level=info msg="Annotations and labels have been set successfully on node: cvm-pclbxnr4"
Dec 01 23:40:03 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:03Z" level=info msg="Starting flannel with backend vxlan"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:04Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-
wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=cvm-pclbxnr4 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=ipt
ables"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:04Z" level=info msg="Server x.x.x.x:6443@RECOVERING*->ACTIVE from successful health check"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:04Z" level=info msg="Server x.x.x.x:6443@UNCHECKED->RECOVERING from successful health check"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:04Z" level=info msg="Server x.x.x.x:6443@UNCHECKED->RECOVERING from successful health check"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.961988  597922 server.go:698] "Successfully retrieved node IP(s)" IPs=["11.0.2.87"]
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: E1201 23:40:04.962097  597922 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort con
nections will be accepted on all local IPs. Consider using `--nodeport-addresses primary`"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.967198  597922 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.967246  597922 server_linux.go:170] "Using iptables Proxier"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.969442  597922 proxier.go:255] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.loc
alhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.969781  597922 server.go:497] "Version info" version="v1.32.8+k3s1"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.969806  597922 server.go:499] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.970710  597922 config.go:329] "Starting node config controller"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.970750  597922 shared_informer.go:313] Waiting for caches to sync for node config
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.970759  597922 config.go:199] "Starting service config controller"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.970772  597922 shared_informer.go:313] Waiting for caches to sync for service config
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.970804  597922 config.go:105] "Starting endpoint slice config controller"
Dec 01 23:40:04 cvm-pclbxnr4 k3s[597922]: I1201 23:40:04.970828  597922 shared_informer.go:313] Waiting for caches to sync for endpoint slice config
Dec 01 23:40:05 cvm-pclbxnr4 k3s[597922]: I1201 23:40:05.271667  597922 shared_informer.go:320] Caches are synced for node config
Dec 01 23:40:05 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:05Z" level=info msg="Server 103.106.59.201:6443@RECOVERING->PREFERRED from successful health check"
Dec 01 23:40:05 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:05Z" level=info msg="Server 64.34.80.169:6443@RECOVERING->PREFERRED from successful health check"
Dec 01 23:40:05 cvm-pclbxnr4 k3s[597922]: I1201 23:40:05.770841  597922 shared_informer.go:320] Caches are synced for service config
Dec 01 23:40:06 cvm-pclbxnr4 k3s[597922]: I1201 23:40:06.071386  597922 shared_informer.go:320] Caches are synced for endpoint slice config
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585475  597922 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network f
or sandbox \"d5821c04bb69ce0f07a3b47929611f0631a1892e4f39321b9f645e5a496f8972\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file
 or directory"
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585551  597922 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup ne
twork for sandbox \"d5821c04bb69ce0f07a3b47929611f0631a1892e4f39321b9f645e5a496f8972\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no su
ch file or directory" pod="gpu-operator/gpu-operator-1759355567-node-feature-discovery-worker-zrl9v"
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585569  597922 kuberuntime_manager.go:1237] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup n
etwork for sandbox \"d5821c04bb69ce0f07a3b47929611f0631a1892e4f39321b9f645e5a496f8972\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no s
uch file or directory" pod="gpu-operator/gpu-operator-1759355567-node-feature-discovery-worker-zrl9v"
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585611  597922 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"gpu-operator-1759355567-n
ode-feature-discovery-worker-zrl9v_gpu-operator(9b81a0f3-30fc-4e66-b669-7a1a0976d3e5)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"gpu-operator-1759355567-node-featur
e-discovery-worker-zrl9v_gpu-operator(9b81a0f3-30fc-4e66-b669-7a1a0976d3e5)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"d5821c04bb69ce0f07a3b47929611f0631a1
892e4f39321b9f645e5a496f8972\\\": plugin type=\\\"flannel\\\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory\"" pod="gpu-operator/gpu-oper
ator-1759355567-node-feature-discovery-worker-zrl9v" podUID="9b81a0f3-30fc-4e66-b669-7a1a0976d3e5"
Dec 01 23:40:14 cvm-pclbxnr4 k3s[597922]: E1201 23:40:14.585499  597922 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network f
or sandbox \"f40220be58300bf11ce6c1b4ca81db711658dfc64cf58848d9444571f20ee307\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file
 or directory"
Dec 01 23:40:14 cvm-pclbxnr4 k3s[597922]: E1201 23:40:14.585569  597922 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup ne
twork for sandbox \"f40220be58300bf11ce6c1b4ca81db711658dfc64cf58848d9444571f20ee307\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no su
ch file or directory" pod="velero/node-agent-8p76m"
Dec 01 23:40:14 cvm-pclbxnr4 k3s[597922]: E1201 23:40:14.585587  597922 kuberuntime_manager.go:1237] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup n
These are the logs from k3s-agent systemd service, where do I specifically check flannel logs?
First, it says this about flannel
Copy code
Dec 01 23:40:03 cvm-pclbxnr4 k3s[597922]: time="2025-12-01T23:40:03Z" level=info msg="Starting flannel with backend vxlan"
and then all this
Copy code
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585475  597922 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network f
or sandbox \"d5821c04bb69ce0f07a3b47929611f0631a1892e4f39321b9f645e5a496f8972\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file
 or directory"
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585551  597922 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup ne
twork for sandbox \"d5821c04bb69ce0f07a3b47929611f0631a1892e4f39321b9f645e5a496f8972\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no su
ch file or directory" pod="gpu-operator/gpu-operator-1759355567-node-feature-discovery-worker-zrl9v"
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585569  597922 kuberuntime_manager.go:1237] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup n
etwork for sandbox \"d5821c04bb69ce0f07a3b47929611f0631a1892e4f39321b9f645e5a496f8972\": plugin type=\"flannel\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no s
uch file or directory" pod="gpu-operator/gpu-operator-1759355567-node-feature-discovery-worker-zrl9v"
Dec 01 23:40:13 cvm-pclbxnr4 k3s[597922]: E1201 23:40:13.585611  597922 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"gpu-operator-1759355567-n
ode-feature-discovery-worker-zrl9v_gpu-operator(9b81a0f3-30fc-4e66-b669-7a1a0976d3e5)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"gpu-operator-1759355567-node-featur
e-discovery-worker-zrl9v_gpu-operator(9b81a0f3-30fc-4e66-b669-7a1a0976d3e5)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"d5821c04bb69ce0f07a3b47929611f0631a1
892e4f39321b9f645e5a496f8972\\\": plugin type=\\\"flannel\\\" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory\"" pod="gpu-operator/gpu-oper
ator-1759355567-node-feature-discovery-worker-zrl9v" podUID="9b81a0f3-30fc-4e66-b669-7a1a0976d3e5"
at last, when I stop the k3s-agent service I get this
Copy code
Dec 01 23:53:31 cvm-pclbxnr4 k3s[606199]: time="2025-12-01T23:53:31Z" level=fatal msg="Failed to start networking: flannel failed to wait for PodCIDR assignment: failed to wait for PodCIDR assignment: timed out waiting for the condition"
c
what version is the server running?
Are you sure you aren’t trying to join an agent that is much newer than the server?
a
Server is also on same version
c
can you show the output of 
kubectl get node -o wide
?
a
Is it possible that cidr range is exhausted for any more agents?
c
it could be. are you using custom cluster cidrs or have more than 255 nodes?
a
Yes, I have a lot of nodes connected to this cluster. Right now I have 270 including three masters
I'm using default config
Will I see any error if this happens? Also, how do I update the cidr? without affecting current workloads
c
oh well then yeah you’re probably out of cidrs
default is 10.42.0.0/16 and each node gets a /24 so that is 255 total nodes
It is not supported to change the cluster or service cidr. You can TRY setting 
--cluster-cidr=10.42.0.0/15
on your server nodes, and restarting them, and it will probably work - but technically this is not supported.
actually that will not work because it’ll run into the service cidr at 10.43.0.0
You will likely need to build a new cluster with a custom CIDR that has enough space available for however many nodes you plan to join.
a
What if I delete few nodes, will the new one come up?
c
Probably yes
Do you have ones that are no longer up to delete?
a
Yes, I will delete some that are not required anymore
105 Views
Open in Slack
PreviousNext
Powered by