https://rancher.com/ logo
Title
c

careful-table-97595

07/14/2022, 12:51 AM
Hello. Tried to upgrade my k3s cluster from v1.22.7+k3s1 to v1.23.8+k3s2 (cluster in HA mode) by stopping k3s, then using k3s-killall script, replacing the k3s binary to the expected version, then restarting k3s daemon. The upgrade itself went well, but a lot of pods were not able to start correctly on the nodes with the new version with this error:
incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
So I reverted back to my original version (v1.22+k3s1) Did I miss something? I haven't found anything relevant in the documentation
Digging into the logs, it seem that this is related to multus CNI...
c

creamy-pencil-82913

07/14/2022, 5:32 AM
Are you using multus with k3s? What version of multus? Sounds like it might need to be upgraded?
f

famous-energy-13283

07/14/2022, 1:07 PM
Yes, multus is part of my tooling. Required for some workloads. multus: 3.7.1 (latest is 3.9 if I'm not mistaking) cni-plugins: 0.9.1 (latest is apparently at 1.1.1) I'll try to upgrade these and will report
c

careful-table-97595

07/15/2022, 4:19 AM
I upgraded multus to the latest version, then upgraded one node to k3s 1.23.8 and the problem came again. So not related to the multus version. But I found the culprit: /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist From k3s 1.22.7 to k3s 1.23.8, it goes from cniVersion 0.3.1 to 1.0.0 Meanwhile, rke2 version 1.23.8 (for a comparison standpoint), if I'm looking at /etc/cni/net.d/, I find out that my canal file (10-canal.conflist) has its cniVersion set to 0.3.1. Looking at the multus version running there (deployed automatically by rke2), I see it at version 3.8 and plugins to 1.0.1 So apparently, k3s changed its flannel configuration somewhere in 1.23, which has an issue. Digging some more, I can find an issue filed there: https://github.com/k3s-io/k3s/issues/5655 Apparently, the fix would be to use flannelcniconf flag to set the desired cni configuration. Still need to understand the usage, though
Oh! @creamy-pencil-82913, you apparently know about this flag, so you'll be able to explain how to use it (saw your name on the issue 🙂 )
c

creamy-pencil-82913

07/15/2022, 4:47 AM
Yes, we (well, me specifically) bumped the CNI config version. The version of containerd and flannel we ship with are both compatible with it. Is there some other component in your system that isn't yet?
Actually sorry, I see you're referencing the option to provide your own config. I was referring to https://github.com/k3s-io/k3s/commit/e6385b234183541776f46542f963a0aaf9cace01 where we bumped the CNI config version. If for some reason you can't get whatever you have on your host updated to be compatible with that, passing your own config is another option.
c

careful-table-97595

07/15/2022, 11:24 AM
Apparently, multus isn't. It supports only these versions: ["0.1.0", "0.2.0", "0.3.0", "0.3.1", "0.4.0"] Validated by creating a network attachment with cniVersion to 1.0.0 on the latest multus version, and it's throwing the error. Guess I won't have the choice to pass my own config file. That should be described in the k3s ReadMe that flannel, as shipped, is incompatible with multus. I must not be the only guy to use it (or maybe because I'm from the telco world, and we're using multus a lot...)
@bland-account-99790 Can you explain how this new flag (FlannelCNIConf) is implemented? How do I use it?
In particular from the command line, is it --flannelcniconf /path/to/my/custom/configfile that I need to use? Lowercase or CamelCase?
c

creamy-pencil-82913

07/15/2022, 5:17 PM
hmm, yeah it looks like multus is still using
<http://github.com/containernetworking/cni|github.com/containernetworking/cni> v0.8.1
which doesn’t support the new version yet. The flag is just
--flannel-cni-conf=/path/to/config
I opened a PR to multus. I haven’t poked at multus before, we’ll see how it goes. https://github.com/k8snetworkplumbingwg/multus-cni/pull/879
f

famous-energy-13283

07/20/2022, 10:23 PM
Meanwhile, the flag did the job. At least, I was able to upgrade. 🙂
c

creamy-pencil-82913

07/20/2022, 11:13 PM
yeah, glad we have that as a work around. the multus team don’t seem to move very quickly.