This message was deleted.
# k3sa
adamant-kite-43734
07/14/2022, 12:51 AMThis message was deleted.
c
careful-table-97595
07/14/2022, 3:53 AMDigging into the logs, it seem that this is related to multus CNI...
c
creamy-pencil-82913
07/14/2022, 5:32 AMAre you using multus with k3s? What version of multus? Sounds like it might need to be upgraded?
f
famous-energy-13283
07/14/2022, 1:07 PMYes, multus is part of my tooling. Required for some workloads.
multus: 3.7.1 (latest is 3.9 if I'm not mistaking)
cni-plugins: 0.9.1 (latest is apparently at 1.1.1)
I'll try to upgrade these and will report
c
careful-table-97595
07/15/2022, 4:19 AMI upgraded multus to the latest version, then upgraded one node to k3s 1.23.8 and the problem came again. So not related to the multus version. But I found the culprit:
/var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist
From k3s 1.22.7 to k3s 1.23.8, it goes from cniVersion 0.3.1 to 1.0.0
Meanwhile, rke2 version 1.23.8 (for a comparison standpoint), if I'm looking at /etc/cni/net.d/, I find out that my canal file (10-canal.conflist) has its cniVersion set to 0.3.1. Looking at the multus version running there (deployed automatically by rke2), I see it at version 3.8 and plugins to 1.0.1
So apparently, k3s changed its flannel configuration somewhere in 1.23, which has an issue.
Digging some more, I can find an issue filed there: https://github.com/k3s-io/k3s/issues/5655
Apparently, the fix would be to use flannelcniconf flag to set the desired cni configuration. Still need to understand the usage, though
careful-table-97595
07/15/2022, 4:21 AMOh! @creamy-pencil-82913, you apparently know about this flag, so you'll be able to explain how to use it (saw your name on the issue 🙂 )
c
creamy-pencil-82913
07/15/2022, 4:47 AMYes, we (well, me specifically) bumped the CNI config version. The version of containerd and flannel we ship with are both compatible with it. Is there some other component in your system that isn't yet?
creamy-pencil-82913
07/15/2022, 5:42 AMActually sorry, I see you're referencing the option to provide your own config. I was referring to https://github.com/k3s-io/k3s/commit/e6385b234183541776f46542f963a0aaf9cace01 where we bumped the CNI config version. If for some reason you can't get whatever you have on your host updated to be compatible with that, passing your own config is another option.
c
careful-table-97595
07/15/2022, 11:24 AMApparently, multus isn't. It supports only these versions:
["0.1.0", "0.2.0", "0.3.0", "0.3.1", "0.4.0"]
Validated by creating a network attachment with cniVersion to 1.0.0 on the latest multus version, and it's throwing the error. Guess I won't have the choice to pass my own config file. That should be described in the k3s ReadMe that flannel, as shipped, is incompatible with multus. I must not be the only guy to use it (or maybe because I'm from the telco world, and we're using multus a lot...)
careful-table-97595
07/15/2022, 11:31 AM@bland-account-99790 Can you explain how this new flag (FlannelCNIConf) is implemented? How do I use it?
careful-table-97595
07/15/2022, 11:47 AMIn particular from the command line, is it
--flannelcniconf /path/to/my/custom/configfile
that I need to use? Lowercase or CamelCase?
c
creamy-pencil-82913
07/15/2022, 5:17 PMhmm, yeah it looks like multus is still using
<http://github.com/containernetworking/cni|github.com/containernetworking/cni> v0.8.1--flannel-cni-conf=/path/to/configcreamy-pencil-82913
07/15/2022, 5:49 PMI opened a PR to multus. I haven’t poked at multus before, we’ll see how it goes.
https://github.com/k8snetworkplumbingwg/multus-cni/pull/879
f
famous-energy-13283
07/20/2022, 10:23 PMMeanwhile, the flag did the job. At least, I was able to upgrade. 🙂
c
creamy-pencil-82913
07/20/2022, 11:13 PMyeah, glad we have that as a work around. the multus team don’t seem to move very quickly.
63 Views