This message was deleted.

Agents expect to be able to connect directly to the server IPs. If the server nodes are actually pods running somewhere, with inaccessible addresses, that will not work.

Following this document: https://docs.k3s.io/datastore/cluster-loadbalancer?ext-load-balancer=HAProxy

Right but even if you do that, the agents still need to be able to connect directly to the servers. The external LB is just used to provide a fixed registration address that the agents initially use to find a server. Once they are connected they switch over to connecting directly to the servers.

Okay that makes sense, so the best option here is to establish L3 connectivity in some way via wireguard or some other solution?

idk that you can run wireguard in a pod, I’ve not tried it.

This is probably not the place to ask, but have you used k0s for a setup like this before? They seem to advertise supporting a setup along these lines, but I was wondering if I would end up in the same situation with that.

no, I have never used k0s

Unfortunately they do already have --disable-agent. These are the flags I am passing:

- server
        - --disable-agent
        - --disable=coredns,servicelb,traefik
        - --tls-san={{tlsSan}}
        - --flannel-backend=none
        - --egress-selector-mode=cluster

yeah, so do you know how

kubectl logs

and

kubectl exec

work? And what the egress-selector is doing?

Correct me if I am wrong but essentially the k8s control plane establishes a websocket? connection to the kubelet on the node which executes the action. As for the egress selector, I am not sure.

right so when you run one of those commands, the apiserver makes a connection to the kubelet to pull logs, or run the command in the pod and pipe output back to the client. This means that the server MUST be able to open a connection to the kubelet. I am guessing that your server pods can’t connect to the agents? K3s includes an embedded egress proxy so that the apiserver can connect to kubelets, using a websocket tunnel connection initiated by the agent. This means that you only need agent -> server connectivity, not agent - server. However, this does mean that the agents need to be able to connect directly to all of the servers, so that every server has the websocket tunnel to use when it needs to talk to that kubelet. The problem you have in your environment, is that servers can’t connect to agents, and agents can’t connect to servers. All anything can connect to is the LB you’ve put in front of the servers.

Thank you for the detailed explanation and help, looks like I have some work to do 🙂