Hi. I'm trying to setup 2 separate RKE2 clusters b...
# rke2
m
Hi. I'm trying to setup 2 separate RKE2 clusters behind an nginx layer 4 load balancer. To be able to use the same load balancer for 2 separate clusters i am using stream_ssl_preread_module to tie each cluster to a separate DNS entry. So far the server nodes can join on the dns name used in the load balancer, i.e https://cluster1.domain.com, and this seems to be because of the use of tls-san. However when i try to join an agent node to https://cluster1.domain.com i keep getting "failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\". I am using ufw and I am certain all ports required are allowed, and since I am able to join a server node this does not seem to be the problem. Does anyone have any suggestions what i should try?
c
Start the agents with --debug. Note that agents need to be able to connect to the servers directly, as well as to the lb. I am also not sure that all of the connections will provide a TLS SNI server name when handshaking.
m
I tried adding debug in the config file for the agent node but I do not get any helpful output there. I'm logging the requests in nginx and it seems like the requests from the agents does not include SNI. This however works on the server. So if i setup a server node on the same machine that i originally want as an agent all works fine. The way I want to setup my load balancer this will not work. Adding my nginx config here to show what i want to achieve. upstream devcluster1 { server 192.168.1.2:6443; } upstream devcluster1_supervisor { server 192.168.1.2:9345; } upstream devcluster2 { server 192.168.2.2:6443; } upstream devcluster2_supervisor { server 192.168.2.2:9345; } map $ssl_preread_server_name $upstream_6443 { dev-cluster1.domain.com devcluster1; dev-cluster2.domain.com devcluster2; } map $ssl_preread_server_name $upstream_9345 { dev-cluster1.domain.com devcluster1_supervisor; dev-cluster2.domain.com devcluster2_supervisor; } server { listen 6443; ssl_preread on; proxy_pass $upstream_6443; } server { listen 9345; ssl_preread on; proxy_pass $upstream_9345; } So the idea is to have 1 load balancer but multiple clusters. Maybe I am over complicating things and this is not a preferred way of setting up multiple clusters. If so, I would be happy to get some input of other options of setting it all up.
c
Yeah that's not something we've ever tested or intentionally support. Generally we recommend a dedicated LB per cluster, using DNS aliases instead of an LB, or just picking one of the servers to always join against.
m
If I would go with picking a server node to always join agains, what would happen if that server node is removed from the cluster. Would the agent still work properly?
c
you’d need to point it at a new server
hence DNS alias being a better choice