This message was deleted.

My impression is that you would want other features a more feature rich cni can deliver. As well as performance not being good enough

I am sure something like Cilium would be a lot better but during testing I came across some issues with cilium paired with K3s that are also documented, so built in flannel is just easier.

Hmm okay. Interesting. Can you point me to those issues please. I would appreciate it.

Sorry I forgot to update this thread. It seems that the problem was due to some issue with Ubuntu 22.04. The latest 24.04 works like a charm and I am using Cilium just fine now! Testing with a 500 node cluster and it's beautiful. Cilium combined with an external datastore (I use Postgres) instead of etcd makes for a very scalable cluster

@straight-businessperson-27680 no worries. What are the documented issues you refer to?

See https://docs.k3s.io/networking/basic-network-options under Cilium section

Hmm okay. So you re-install on the same node? We switch out nodes … introducing new ones and then removing old ones.

Can you tell me more? What do you do exactly?

When we e.g. upgrade K3s we have a flow that ensures that new nodes are introduced ( with the never version ) >> old nodes are removed by draining and so forth. This even works with stateful workloads. We run Postgresql, Kafka and so forth ( Longhorn is the underlying CSI ). So we basically treat workers as cattle not pets.

nice, how do you automate it?

By having build a project that makes it possible. We’re working on open sourcing it.

we have solved this orchestration issue with bosh (old but capable vm multi cloud orchestration solution), with a port of k3s https://github.com/orange-cloudfoundry/k3s-wrapper-boshrelease

Interesting. When you say orchestration issue solved .. you’re referring to the bootstrapping of K3s here?