https://rancher.com/ logo
Powered by
b

big-jordan-45387

01/12/2023, 6:50 PM
hi, I am trying to create a new cluster in rancher using rke2 (I am very new to rancher), deployment fails because cattler-cluster-agent can't resolve the rancher server dns. I kept digging into this and I found this error in calico:
Copy code
2023-01-12 18:18:10.939 [INFO][148571] felix/ipsets.go 309: Retrying after an ipsets update failure... family="inet"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=1 setID="all-ipam-pools"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=1 setID="masq-ipam-pools"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=6 setID="this-host"
2023-01-12 18:18:10.941 [INFO][148571] felix/ipsets.go 779: Doing full IP set rewrite family="inet" numMembersInPendingReplace=0 setID="all-vxlan-net"
2023-01-12 18:18:10.943 [WARNING][148571] felix/ipsets.go 713: Failed to complete ipset restore, IP sets may be out-of-sync. closeErr=<nil> commitErr=<nil> family="inet" flushErr=<nil> input="create cali40all-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4t32 hash:net family inet maxelem 1048576\nadd cali4t32 10.42.0.0/16\nswap cali40all-ipam-pools cali4t32\ndestroy cali4t32\ncreate cali40masq-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4t33 hash:net family inet maxelem 1048576\nadd cali4t33 10.42.0.0/16\nswap cali40masq-ipam-pools cali4t33\ndestroy cali4t33\ncreate cali40this-host hash:ip family inet maxelem 1048576\ncreate cali4t34 hash:ip family inet maxelem 1048576\nadd cali4t34 127.0.0.0\nadd cali4t34 127.0.0.1\nadd cali4t34 10.100.32.62\nadd cali4t34 148.187.113.235\nadd cali4t34 172.17.0.1\nadd cali4t34 10.42.224.64\nswap cali40this-host cali4t34\ndestroy cali4t34\ncreate cali40all-vxlan-net hash:net family inet maxelem 1048576\ncreate cali4t35 hash:net family inet maxelem 1048576\nswap cali40all-vxlan-net cali4t35\ndestroy cali4t35\nCOMMIT\n" processErr=exit status 1 stderr="ipset v7.11: Error in line 1: Kernel error received: set type not supported\n" stdout="" writeErr=<nil>
2023-01-12 18:18:10.943 [WARNING][148571] felix/ipsets.go 340: Failed to update IP sets. Marking dataplane for resync. error=exit status 1 family="inet"
My hosts are suse enterprise 15.3 and rancher version I am using is v2.7.0 and v1.24.8+rke2r1
Not sure this is relevant...
Copy code
nid003207:~ # ipset
ipset v6.36: No command specified.
Try `ipset help' for more information.
c

creamy-pencil-82913

01/12/2023, 7:22 PM
Copy code
Kernel error received: set type not supported
Looks like your kernel is too old?
b

big-jordan-45387

01/12/2023, 7:22 PM
it is suse enterprise 15.3
but yeah I am checking if I can update ipset to 7.11 or so
c

creamy-pencil-82913

01/12/2023, 7:22 PM
its not the ipset binary thats the problem
as it says, the kernel doesn’t support the set type that calico wants to use
either that or you’re missing kernel modules, not sure which
can you 
modprobe xt_set
?
b

big-jordan-45387

01/12/2023, 7:25 PM
I think so, at leat it does not compalin
c

creamy-pencil-82913

01/12/2023, 7:25 PM
does that fix the error?
b

big-jordan-45387

01/12/2023, 7:25 PM
let me check
should I kill the calico pod?
c

creamy-pencil-82913

01/12/2023, 7:26 PM
sure
b

big-jordan-45387

01/12/2023, 7:27 PM
same error
c

creamy-pencil-82913

01/12/2023, 7:28 PM
interesting
what kernel version are you on?
We test on SLES 15.3 and 15.4 and have not run into that issue
b

big-jordan-45387

01/12/2023, 7:29 PM
Copy code
nid003207:~ # uname -r
5.3.18-150300.59.68_11.0.76-cray_shasta_c
ppl from calico are saying I should update the ipset in my kernel to match the one calico expects
c

creamy-pencil-82913

01/12/2023, 7:31 PM
that looks like a custom kernel
I suspect that whoever built it didn’t enable all the correct ipset options
If you’re running a custom kernel that’s on you (or your sysadmin) to ensure that it has all the same bits as the default SLES kernel, otherwise things will break
b

big-jordan-45387

01/12/2023, 7:33 PM
yeah
true
c

creamy-pencil-82913

01/12/2023, 7:38 PM
you might see if canal fares any better but I suspect you’ll run into the same problem
b

big-jordan-45387

01/12/2023, 7:39 PM
mmm I tried cilium and had the same issue, but yeah I can try canal also
first I am trying to update the ipset and see if I get things working
c

creamy-pencil-82913

01/12/2023, 7:40 PM
its really the kernel and not the binary…
b

big-jordan-45387

01/12/2023, 10:08 PM
with canal works
it is weird because is also shows the same error message
so this tells me the issues is not with ipset when using calico
ipset 7.17 requires kernel >= 3.11 https://ipset.netfilter.org/install.html so my kernel should be ok to run calico
c

creamy-pencil-82913

01/12/2023, 11:51 PM
right but there are different ipset hash algs and set types, the ipset binary and kernel have to both support the ones that the CNI is trying to use.
the error suggests that the ipset command is OK with it, but the kernel is not
b

big-jordan-45387

01/12/2023, 11:55 PM
ok, let me keep investigating
its is weird canal works however it uses an older version of calico which uses ipset6
45 Views
Powered by