Hi! I need some basic support on k3d node creation...
# k3d
d
Hi! I need some basic support on k3d node creation. I've created a node, used
-c
and
-t
parameters so the node would join my existing cluster. I can see it joined the cluster successfully, but I get some pods created in my new node with
SysctlForbidden
status. They're called
svclb-**traefik-**
, so I believe it's the network setup trying to replicate svcs entrypoints and failing. If I describe some of these pods, I see a Warning event from kubelet with message
forbidden sysctl: "net.ipv4.ip_forward" not whitelisted
. K3D is running in my local PC, I'm using it to learn how to join and leave clusters, maybe to run k3s in HA etc. The pc uses arch linux up-to-date and docker 28.x. I see that
net.ipv4.ip_forward
is already 1 in this machine, but I see no references to whitelists anywhere. How can I debug this situation?
w
Hey! I cannot help you on the host level setup, but I can tell you that the
svclb*
pods are from the K3s built-in service loadbalancer implementation. That's a way to provide services with
type: LoadBalancer
in K3s. Basically it deploys proxy pods to the nodes (k3s containers in the case of k3s) with
hostPort
bindings.
d
Hi @wide-garage-9465! Thanks
w
I guess you need to tell the Kubelet/K3s that it's ok to use that sysctl via
--allowed-unsafe-sysctls=net.ipv4.ip_forward
(passed in as a k3s-arg). I wonder why this would be required though. How did you create the cluster and how did you join the node? Can you paste the commands here, please?
d
Yes! I created the cluster in an empty ubuntu 24 server, few weeks ago, using default setup syntax:
curl -sfL <https://get.k3s.io> | sh -
. Today I've installed k3d in my local PC using
wget -q -O - <https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh> | bash
, then I ran
k3d node create k3d-arch-01 -c <https://url-to-server:6443> -t very-l0ng-tok3n
w
Ah, I didn't get that part and thought the existing setup was also in k3d. FYI: distributed clusters with k3d are not really supported (with some extra config you can certainly make them work though - we just haven't gotten around to making it automatic in k3d). I'd try creating the node with
--k3s-arg '--kubelet-arg=allowed-unsafe-sysctls=net.ipv4.ip_forward'
d
I'll try it out and report in here. I also got a little confuse when you showed the arg for the first time, like 'in current cluster or in new node?'. It's clear now 😃
--k3s-arg
is documented in k3d.io docs, but my installed CLI says
unknown flag: --k3s-arg
. Running
k3d node create --help
also does not show this flag in list. There's a second flag also not available in my system, it's
--runtime-ulimit
. I'm using k3d v5.0.0
wow, I'm really outdated
w
Yeah, quite a bit 😅
d
I see that k3d docs allow you to select the version, I'll do it. I just didn't expect the default syntax to provide me 5.0.0
edit: I'm pretty sure I copied the wrong example from the docs and if I've used the default syntax it would be the most recent release, actually
I tried creating the node again, still without the arg, to be sure that the issue I was facing wasn't just using a very old version of k3d. Now I've got the following error message: failed to add 1 node(s) to the remote cluster 'https://local-ip:6443': failed to add one or more nodes: failed to run node 'k3d-k3d-arch-02-0': failed to start node 'k3d-k3d-arch-02-0': failed to enable k3d fixes: Cannot enable DNS fix, as Host Gateway IP is missing!
@wide-garage-9465 I verified that this message applies both to using --k3s-arg and not using it, so it probably refers to some network issue I have in my lan
w
Yeah that's something different. Probably related to trying to connect to a remote K3s cluster. I would have to try it, but here are quite a few open issues open for distributed clusters with k3d.
d
Good point, I'll take a look at the open issues too