On the rke2 agent node, the rke2 supervisor client load balancer works fine (port 6444) but the API server lb (port 6443) accepts the connection and does nothing

OK well finally made this work. rke2 agent's api lb was simply not working. For anyone else who might discover this and want a solution here is what I did: I copied a working kubeconfig over and started a kubectl proxy to the cluster

kubectl --kubeconfig cluster.yaml proxy

I copied the api server certificate and key

/var/lib/rancher/rke2/server/serving-kube-apiserver.(crt|key)

over to the agent node and configured stunnel to listen on 127.0.0.1:6443 and proxy to 127.0.0.1:8001 where kubectl proxy was listening. I used docker to do this but sadly I missed retaining the command. I then made the kubeconfig files under /var/lib/rancher/rke2/agent immutable

chmod +i *.kubeconf

OK well finally made this work. rke2 agent's api lb was simply not working. For anyone else who might discover this and want a solution here is what I did. Sorry this is more of a retrospective than a step-by-step. I copied a working kubeconfig over and started a kubectl proxy to the cluster

kubectl --kubeconfig cluster.yaml proxy

I changed the port that rke2 agent uses for load balancing to 8444 by editing

/etc/systemd/system/multi-user.target.wants/rke2-agent.service

to add

--lb-server-port 8444

This causes rke2 agent's non-working LB to avoid grabbing ports 6444/6443 as we want a working api server on 6443. I copied the api server certificate and key

/var/lib/rancher/rke2/server/serving-kube-apiserver.(crt|key)

over to the agent node and configured stunnel to listen on 127.0.0.1:6443 and proxy to 127.0.0.1:8001 where kubectl proxy was listening. I used docker to do this but sadly I missed retaining the command. I then made the kubeconfig files under /var/lib/rancher/rke2/agent immutable so that rke2 agent would not clobber them when starting up and change them to a different port.

chmod +i *.kubeconfig

When rke2 agent complained that it could not open them, I had to remove the immutable flag and immediately put it back, but this allowed the rke2 agent service to start and all of its containers to have connectivity to the api server.