https://rancher.com/ logo
#k3s
Title
# k3s
w

wonderful-spring-28306

08/19/2022, 7:59 AM
Hey @creamy-pencil-82913 Following from this https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ My aim is to create a k3s cluster where the control-plane nodes are not accessible with
kubectl get no
Research shows me that I should start the kubernetes process with
--register-node=false
Is this possible with k3s or do you have a better suggestion on how I could achieve this?
c

creamy-pencil-82913

08/19/2022, 5:09 PM
You can try starting the servers with --disable-agent to not run containerd or kubelet, but it's unsupported. The kubelet won't really be functional without a node object anyway. Also note that if you're using embedded etcd, we need the node object to manage the etcd cluster membership, so that probably won't work right.
w

wonderful-spring-28306

08/19/2022, 6:10 PM
Understood, so if I'm to use --disable agent I cannot have multiple control plane nodes
c

creamy-pencil-82913

08/19/2022, 9:07 PM
With embedded etcd, yes. You can still use multiple servers with external SQL when disabling the agent, as that does not require node resources to coordinate.
w

wonderful-spring-28306

08/19/2022, 9:19 PM
Got it, I'll try with disable agent
Hey @creamy-pencil-82913 I was able to get k3s 1.21 working with
--register-node=false
. Using
kilo CNI
and
kgctl
tool I was also able to get metrics server working. I'm wondering if I can do the same with etcd.
Hey @creamy-pencil-82913 I was able to spin up a HA cluster with 3 controllers and
--disabled-agent
& embedded etcd. etcdctl shows correct etcd membership. As before my setup allows cluster & pod ip to be resolvable and reachable from the
--disabled-agent
controller nodes. Are there any gotchas I need to look out for? k3s 1.21 , kilo cni & kgctl tool on controller nodes.
l

late-needle-80860

08/28/2022, 5:32 PM
Interested @wonderful-spring-28306. Why are you doing this? What’s the goal? And also why the old version of K3s?
w

wonderful-spring-28306

08/28/2022, 5:36 PM
Hey @late-needle-80860
I'm using 1.21 because this has not yet been backported to it https://github.com/k3s-io/k3s/pull/5382
l

late-needle-80860

08/28/2022, 5:39 PM
And your setup do not support that change?
w

wonderful-spring-28306

08/28/2022, 5:41 PM
The ultimate goal is to have controllers without node objects hence not accesible via kubectl and impossible to schedule pods. This can obviously be achieved by using taints but having no node objects in controllers provides better isolation.
l

late-needle-80860

08/29/2022, 7:47 AM
Okay sure. However, it sounds like that’s possible based on Brandon’s response.
w

wonderful-spring-28306

08/29/2022, 7:49 AM
Yes. I verified that it's possible for a single controller however metrics-server was broken. I fixed that with
kgctl
As per @creamy-pencil-82913 HA with embedded etcd should not work with
--disabled-agent
However I have also got that working so I want to know if I'm missing anything.
3 Views