Someone on here having experience with the joining of K3s wo rancher-users #k3s

Title

late-needle-80860

10/07/2022, 10:40 AM

Someone on here having experience with the joining of K3s worker nodes in cordoned fashion … for different reasons we wish that X joining

worker

node joins cordoned in order to different processes to complete in due time before regular workloads starts piling in on the new

worker

. Some of these processes might be/are: • the configuration of

containerd

for a private self-hosted registry • Longhorn bootstrapping … and storage space setup … which needs to be fully up and ready before potential persistent storage needing regular workloads starts appearing

Anyone?

creamy-pencil-82913

10/07/2022, 5:25 PM

no, you can’t create it as cordoned. Cordoned is a combination of a taint (which you can register the node with) and setting spec.unschedulable to true on the node object, which you cannot do.

Does registering it with taints not do enough for you?

also, if you’re using registries.yaml, then containerd is configured before the kubelet starts. And PVCs will just stay pending until Longhorn is ready

None of these seems like things that would require a node to be cordoned, they should just resolve themselves as the cluster comes up normally.

late-needle-80860

10/07/2022, 8:05 PM

Great input love it … will need to think more about it. We’re using

registries.yaml

indeed. That’s injected over

SSH

. Haven’t found a better way of automating that as of yet. But, it’s injected after the node becomes ready .. but maybe we should “just” inject

registries.yaml

earlier … if possible.

Indeed a taint is an option as well. Would just really like to avoid having to tell consumers of the platform to taint with this or that …

creamy-pencil-82913

10/07/2022, 8:10 PM

wait so you’re installing k3s, waiting for the node to become ready, then ssh-ing in to drop a registries.yaml and restart k3s?

😊 1

cordon is taint + unschedulable. It’s worse than a taint, in terms of things to ask customers to do.

why can’t you drop the registries.yaml at the same time you’re installing k3s, before starting it?

late-needle-80860

10/08/2022, 7:36 AM

I might just be very able to do that … so I’ll certainly try that. As that would alleviate some unnecessarily complex logic.

cordon is taint + unschedulable. It’s … sure … We control the process completely though. But, yes in the hopefully not to distant future we’re going to deliver auto-scaling on the node side - and that process needs to be as streamlined as possible.

So having to cordon: taint+unschedulable … is not preferred I agree

Thinking about it I’m quite sure that we inject the

registries.yaml

file after K3s is up is because the way the tool we use for bootstrapping

K3s

clusters injects files before

K3s

is fully up .. that means that the

--data-dir

that we set is not ready at file injection time - e.g. when injecting into the

./manifests

folder - I’ve been assuming that was the same for the

registries.yaml

file where that needs to live (

etc/rancher/k3s/server

) if I remember the path correctly. I’ll see if that flow is an issue for the

registries.yaml

file. The HCI integration/bootstrapping Kubernetes tool we use, if interested, is https://github.com/karmab/kcli

2 Views

#k3s Join Slack