hi I am trying to create a new cluster in rancher using rke2 rancher-users #general

Title

big-jordan-45387

01/12/2023, 6:50 PM

hi, I am trying to create a new cluster in rancher using rke2 (I am very new to rancher), deployment fails because cattler-cluster-agent can't resolve the rancher server dns. I kept digging into this and I found this error in calico:

2023-01-12 18:18:10.939 [INFO][148571] felix/ipsets.go 309: Retrying after an ipsets update failure... family="inet"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=1 setID="all-ipam-pools"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=1 setID="masq-ipam-pools"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=6 setID="this-host"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=0 setID="all-vxlan-net"
2023-01-12 18:18:10.943 [WARNING][148571] felix/ipsets.go 713: Failed to complete ipset restore, IP sets may be out-of-sync. closeErr=<nil> commitErr=<nil> family="inet" flushErr=<nil> input="create cali40all-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4t32 hash:net family inet maxelem 1048576\nadd cali4t32 10.42.0.0/16\nswap cali40all-ipam-pools cali4t32\ndestroy cali4t32\ncreate cali40masq-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4t33 hash:net family inet maxelem 1048576\nadd cali4t33 10.42.0.0/16\nswap cali40masq-ipam-pools cali4t33\ndestroy cali4t33\ncreate cali40this-host hash:ip family inet maxelem 1048576\ncreate cali4t34 hash:ip family inet maxelem 1048576\nadd cali4t34 127.0.0.0\nadd cali4t34 127.0.0.1\nadd cali4t34 10.100.32.62\nadd cali4t34 148.187.113.235\nadd cali4t34 172.17.0.1\nadd cali4t34 10.42.224.64\nswap cali40this-host cali4t34\ndestroy cali4t34\ncreate cali40all-vxlan-net hash:net family inet maxelem 1048576\ncreate cali4t35 hash:net family inet maxelem 1048576\nswap cali40all-vxlan-net cali4t35\ndestroy cali4t35\nCOMMIT\n" processErr=exit status 1 stderr="ipset v7.11: Error in line 1: Kernel error received: set type not supported\n" stdout="" writeErr=<nil>
2023-01-12 18:18:10.943 [WARNING][148571] felix/ipsets.go 340: Failed to update IP sets. Marking dataplane for resync. error=exit status 1 family="inet"

My hosts are suse enterprise 15.3 and rancher version I am using is v2.7.0 and v1.24.8+rke2r1

Not sure this is relevant...

nid003207:~ # ipset
ipset v6.36: No command specified.
Try `ipset help' for more information.

creamy-pencil-82913

01/12/2023, 7:22 PM

Kernel error received: set type not supported

Looks like your kernel is too old?

big-jordan-45387

01/12/2023, 7:22 PM

it is suse enterprise 15.3

but yeah I am checking if I can update ipset to 7.11 or so

creamy-pencil-82913

01/12/2023, 7:22 PM

its not the ipset binary thats the problem

as it says, the kernel doesn’t support the set type that calico wants to use

either that or you’re missing kernel modules, not sure which

https://github.com/kubernetes/minikube/pull/1697

can you

modprobe xt_set

big-jordan-45387

01/12/2023, 7:25 PM

I think so, at leat it does not compalin

creamy-pencil-82913

01/12/2023, 7:25 PM

does that fix the error?

big-jordan-45387

01/12/2023, 7:25 PM

let me check

should I kill the calico pod?

creamy-pencil-82913

01/12/2023, 7:26 PM

sure

big-jordan-45387

01/12/2023, 7:27 PM

same error

creamy-pencil-82913

01/12/2023, 7:28 PM

interesting

what kernel version are you on?

We test on SLES 15.3 and 15.4 and have not run into that issue

big-jordan-45387

01/12/2023, 7:29 PM

nid003207:~ # uname -r
5.3.18-150300.59.68_11.0.76-cray_shasta_c

ppl from calico are saying I should update the ipset in my kernel to match the one calico expects

creamy-pencil-82913

01/12/2023, 7:31 PM

that looks like a custom kernel

I suspect that whoever built it didn’t enable all the correct ipset options

see: https://github.com/weaveworks/weave/issues/3062#issuecomment-315335046

If you’re running a custom kernel that’s on you (or your sysadmin) to ensure that it has all the same bits as the default SLES kernel, otherwise things will break

big-jordan-45387

01/12/2023, 7:33 PM

yeah

true

creamy-pencil-82913

01/12/2023, 7:38 PM

you might see if canal fares any better but I suspect you’ll run into the same problem

big-jordan-45387

01/12/2023, 7:39 PM

mmm I tried cilium and had the same issue, but yeah I can try canal also

first I am trying to update the ipset and see if I get things working

creamy-pencil-82913

01/12/2023, 7:40 PM

its really the kernel and not the binary…

big-jordan-45387

01/12/2023, 10:08 PM

with canal works

it is weird because is also shows the same error message

so this tells me the issues is not with ipset when using calico

ipset 7.17 requires kernel >= 3.11 https://ipset.netfilter.org/install.html so my kernel should be ok to run calico

creamy-pencil-82913

01/12/2023, 11:51 PM

right but there are different ipset hash algs and set types, the ipset binary and kernel have to both support the ones that the CNI is trying to use.

the error suggests that the ipset command is OK with it, but the kernel is not

big-jordan-45387

01/12/2023, 11:55 PM

ok, let me keep investigating

its is weird canal works however it uses an older version of calico which uses ipset6

30 Views

#general Join Slack