Hi, what would be a best practice to manage the Helm Charts (and Helm Chart Configs) manifests used by RKE2? Manifests defined in the /var/lib/rancher/rke2/server/manifests are applied to the cluster. But what if you have multiple masters, should you define the manifest on all master nodes or just on a single node? And how deals the Helm Controller then when two different manifests (with the same name and namespace used) on two different nodes? My best bet is to write the manifests on all nodes, but not sure on that (and I can’t find any documentation on this)

The trick here is that you have to ensure that leader gets the right configuration and probably the best way to accomplish this is replicating the configuration on all master nodes because leadership can change...

Thank you, so every node deals with its ‘own’ configuration even if this can cause some conflicts between configuration of different nodes

Well I think does not really true. When leader restarts and read its config file changes, they will be applied. That's why you have to keep files synced as leadership may change in your cluster and misconfigurations may appear if files are differ from node to node.

Is there actually a leader node within RKE2? As far as I understand there is only an initial server node. After joining a second and third node all master nodes are equal. So when having a configuration (both /etc/rancher/rke2/config or /var/lib/rancher/rke2/server/manifests) on master 1 which differs from master 2, it looks like just the most recent configuration is applied. But either way, configuration should be in sync on all the master nodes.

master will stay equal until you do some changes in their configs. If you want to upgrade a configuration you will change files and may get unsynchronized...

Most distributed systems have a leader election system so they can use the leader to be a checkpoint to avoid race conditions or a place to tally votes to decide things. Programming follows the programmer's thought, and having a "leader" to make decisions and validate ordering of events fits pretty well with human thought. I'd expect etcd to have a leader and I seem to recall kube-scheduler & kube-controller-manager both had leader elections as well (I was having a problem where the leader election for both of those was failing, so I couldn't get new pods launched and I didn't get updates for how many pods were running, and the logs for all were just spinning trying to get leader election).

manifest application is not leader elected. All servers will apply their manifests. So either pick one node to put your manifests on, or make sure they’re in sync on all of then.

Not sure if I am wrong, but helmchartconfig only applies on restart. Am I right?

all manifests are applied whenever they change on disk

Thank you @creamy-pencil-82913, can you advise on a common best practice? Like deploying all manifests on all servers (via Ansible or some other tool) or keep them just on a single server?

@creamy-pencil-82913, if changes are applied whenever they appear on disk, what happens when different files exist on different nodes?. There should be some mechanism to manage this situation as it can be critical.

From what was said, my guess is all manifests basically get a kubectl apply. Now if they only get that when they change on disk or periodically check and re-apply if things diverges isn't clear, though my guess would be the first.

what happens when different files exist on different nodes?

If there are different files on different nodes, then you get the combination of the files on all the different nodes deployed to the cluster. If you have the same files on all the nodes, but with different content. then that’s bad. Don’t do that.

Then it seems easier to maintain your own system helm charts outside of this RKE2 automation and this will ensure your configuration is correctly applied. Now it seems the best approach for me.

yeah I probably wouldn’t maintain too much complicated stuff via the manifests. It’s mostly used when you have a core component with a fairly static configuration that you need applied right when the cluster comes up. Cloud provider, ingress controller, et cetera.

In fact my question was related to ingress controller reconfiguration for special needs. Probably I will remove the built-in ingress controller and apply a more config-controlled one. Thanks.