I may have found the issue. I think Rancher on k3s does not work with Harvester since the TLS certs are in a different location. They must be hard-coded or something on the cattle-cluster-agent deployment which is what hooks up rancher to harvester. Found the real problem. More of a workaround. When you deploy Rancher you have to add this option to the helm chart

--set agentTLSMode=system-store

So it should look something like this

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=HOSTNAME.EXAMPLE.COM \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=certs@hotmail.ca \
  --set letsEncrypt.ingress.class=nginx \
  --set agentTLSMode=system-store

For the life of my I can't figure out the proper way to add CA for TLS to get trusted and work in the normal way. This workaround just skips that validation. This now works on both rke2 and k3s clusters, so my initial hunch was incorrect.