I completed a similar setup this past week, but with K3S instead of RKE2. I am using an NGINX instance performing SSL off-load and HTTP to HTTPS redirection. But I ran into a similar issue. I did manage to resolve it. Here are some of my notes:
• I did not use the "privateCA=true" or the "ingress.tls.source=secret". I only needed the "tls=external"
• I did not install Cert Manager (made this mistake the first time around, and reinstalled)
• For the K3S included Traefik instance, I had to enable HTTPS header inspection (ref:
https://github.com/rancher/rancher/issues/31071 and:
https://github.com/rancher/rancher/issues/35088 )
• I did configure my external NGINX instance to forward the necessary headers per the Rancher docs
• I had to chain my intermediate cert on the NGINX instance.
Ultimately, it was the intermediate cert chain on my NGINX external LB that was my issue. Since I am using a public cert and DNS name, I ran the DNS through
https://www.ssllabs.com/ssltest/. I adjusted the certificate chain (order is important) until the SSL test website gave me an "A" rating.
Once that was complete, I SSH'd into the node on the downstream cluster that Rancher was waiting for a check-in from. I then ran the /usr/local/custom-scripts/install.sh file, and the agent successfully downloaded and connected.
Hope that helps.