Does registering it with taints not do enough for you?

also, if you’re using registries.yaml, then containerd is configured before the kubelet starts. And PVCs will just stay pending until Longhorn is ready

None of these seems like things that would require a node to be cordoned, they should just resolve themselves as the cluster comes up normally.

Indeed a taint is an option as well. Would just really like to avoid having to tell consumers of the platform to taint with this or that …

cordon is taint + unschedulable. It’s worse than a taint, in terms of things to ask customers to do.

why can’t you drop the registries.yaml at the same time you’re installing k3s, before starting it?

cordon is taint + unschedulable. It’s … sure … We control the process completely though. But, yes in the hopefully not to distant future we’re going to deliver auto-scaling on the node side - and that process needs to be as streamlined as possible.

So having to cordon: taint+unschedulable … is not preferred I agree

Thinking about it I’m quite sure that we inject the

registries.yaml

file after K3s is up is because the way the tool we use for bootstrapping

K3s

clusters injects files before

K3s

is fully up .. that means that the

--data-dir

that we set is not ready at file injection time - e.g. when injecting into the

./manifests

folder - I’ve been assuming that was the same for the

registries.yaml

file where that needs to live (

etc/rancher/k3s/server

) if I remember the path correctly. I’ll see if that flow is an issue for the

registries.yaml

file. The HCI integration/bootstrapping Kubernetes tool we use, if interested, is https://github.com/karmab/kcli