Hello. Tried to upgrade my k3s cluster from v1.22.7+k3s1 to v1.23.8+k3s2 (cluster in HA mode) by stopping k3s, then using k3s-killall script, replacing the k3s binary to the expected version, then restarting k3s daemon. The upgrade itself went well, but a lot of pods were not able to start correctly on the nodes with the new version with this error:

incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"

So I reverted back to my original version (v1.22+k3s1) Did I miss something? I haven't found anything relevant in the documentation

Are you using multus with k3s? What version of multus? Sounds like it might need to be upgraded?

Yes, multus is part of my tooling. Required for some workloads. multus: 3.7.1 (latest is 3.9 if I'm not mistaking) cni-plugins: 0.9.1 (latest is apparently at 1.1.1) I'll try to upgrade these and will report

I upgraded multus to the latest version, then upgraded one node to k3s 1.23.8 and the problem came again. So not related to the multus version. But I found the culprit: /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist From k3s 1.22.7 to k3s 1.23.8, it goes from cniVersion 0.3.1 to 1.0.0 Meanwhile, rke2 version 1.23.8 (for a comparison standpoint), if I'm looking at /etc/cni/net.d/, I find out that my canal file (10-canal.conflist) has its cniVersion set to 0.3.1. Looking at the multus version running there (deployed automatically by rke2), I see it at version 3.8 and plugins to 1.0.1 So apparently, k3s changed its flannel configuration somewhere in 1.23, which has an issue. Digging some more, I can find an issue filed there: https://github.com/k3s-io/k3s/issues/5655 Apparently, the fix would be to use flannelcniconf flag to set the desired cni configuration. Still need to understand the usage, though

Yes, we (well, me specifically) bumped the CNI config version. The version of containerd and flannel we ship with are both compatible with it. Is there some other component in your system that isn't yet?

Apparently, multus isn't. It supports only these versions: ["0.1.0", "0.2.0", "0.3.0", "0.3.1", "0.4.0"] Validated by creating a network attachment with cniVersion to 1.0.0 on the latest multus version, and it's throwing the error. Guess I won't have the choice to pass my own config file. That should be described in the k3s ReadMe that flannel, as shipped, is incompatible with multus. I must not be the only guy to use it (or maybe because I'm from the telco world, and we're using multus a lot...)

hmm, yeah it looks like multus is still using

<http://github.com/containernetworking/cni|github.com/containernetworking/cni> v0.8.1

which doesn’t support the new version yet. The flag is just

--flannel-cni-conf=/path/to/config

Meanwhile, the flag did the job. At least, I was able to upgrade. 🙂

yeah, glad we have that as a work around. the multus team don’t seem to move very quickly.