https://rancher.com/ logo
Join Slack
Powered by
Hello, Understood this is the right place for Flan...
# k3s
a
Hello, Understood this is the right place for Flannel questions. Using Flannel 0.27.2 If we run long tests that use the same cluster for a long time, we get eventually sometimes this error, i believe we dont have 256 pods at the same time, but i didnt double check that, is it a known issue ? can we add some debug commands to see if indeed there is IP leak ? the config we are using is 16 bit network, and the default 24 bit per node https://github.com/kubevirt/kubevirtci/pull/1506/commits/c098fa396d48ee3ca1dbe90962819a536b3a1dfb#diff-dad13ceea48beb5dde[…]d22d442529fb2aefcfc7ac63 We might change meanwhile to have more bits per node so it might WA this problem (but best to solve it of course) Will attach the log in the thread Thanks
Copy code
14m         Warning   FailedCreatePodSandBox   pod/secondary-dns-5fc6686967-t97vf   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_secondary-dns-5fc6686967-t97vf_cluster-network-addons_9b165a5c-539f-41d4-9781-1f5ae2cb3311_0(5c39f1e51a02a48e2d99e0670c449379f2d45c9c631971d96f078b55931c3402): error adding pod cluster-network-addons_secondary-dns-5fc6686967-t97vf to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5c39f1e51a02a48e2d99e0670c449379f2d45c9c631971d96f078b55931c3402" Netns:"/var/run/netns/b30d651d-76fb-4e01-bcae-b162095eeba8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=cluster-network-addons;K8S_POD_NAME=secondary-dns-5fc6686967-t97vf;K8S_POD_INFRA_CONTAINER_ID=5c39f1e51a02a48e2d99e0670c449379f2d45c9c631971d96f078b55931c3402;K8S_POD_UID=9b165a5c-539f-41d4-9781-1f5ae2cb3311" Path:"" ERRORED: error configuring pod [cluster-network-addons/secondary-dns-5fc6686967-t97vf] networking: [cluster-network-addons/secondary-dns-5fc6686967-t97vf/9b165a5c-539f-41d4-9781-1f5ae2cb3311:cbr0]: error adding container to network "cbr0": plugin type="flannel" failed (add): failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254...
might found out why it happens info in the issue
b
thanks
a
Gm What do you think about the idea there to add GC ?
With AI i can create something initial if desired
b
I actually would like first to understand the reasons why this is happening
and when we understand why, if it is impossible to fix, then think of a GC
a
np thanks there are 2 reasons mentioned in the issue if you are interested 1. node reboot, causing unsync between pods and ip lease files (no cmdDel will come for those ever), the cbr0 folder that is used by default is non volatile 2. multus is uninstalled / reinstalled while deleting pods, so cmdDel is lost, hence ip zombie is left
b
Some months ago I tried with 1 and I could not replicate it. IIRC, there was some GC going on that would clean up the files. I'd like to try 2
@full-afternoon-93056 @plain-byte-79620 didn't we test 1 together?
a
if you need a setup that reproduce 2 easily please let me know we have such CI tests, that can be run locally
[1] seems to happen here as well https://github.com/canonical/microk8s/issues/5267 if you are interested, but i guess you saw it
2 Views
Open in Slack
PreviousNext
Powered by