Rancher v2.7.0 with downstream RKE2 cluster. Not able to import an orphaned cluster into a new instance of Rancher. Import Existing -> Import any Kubernetes cluster. After issuing the import command on the cluster several pods go into CrashLoop and do not recover:

/var/lib/rancher/rke2/bin/kubectl         --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f <https://rancher75182.senode.dev/v3/import/xhctfcnbbt56xvxh6jptq7lzvpw9svd2drkbj5pvm466t5r7zlplqv_c-m-zqcvzlgn.yaml>
<http://clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver|clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver> unchanged
<http://clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master|clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master> unchanged
namespace/cattle-system unchanged
serviceaccount/cattle unchanged
<http://clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding|clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding> unchanged
secret/cattle-credentials-ad9a794 created
<http://clusterrole.rbac.authorization.k8s.io/cattle-admin|clusterrole.rbac.authorization.k8s.io/cattle-admin> unchanged
Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: <http://beta.kubernetes.io/os|beta.kubernetes.io/os> is deprecated since v1.14; use "<http://kubernetes.io/os|kubernetes.io/os>" instead
deployment.apps/cattle-cluster-agent configured
service/cattle-cluster-agent unchanged

NAMESPACE             NAME                                                    READY   STATUS             RESTARTS      AGE
calico-system         calico-kube-controllers-f75c97ff6-fvb66                 1/1     Running            0             19m
calico-system         calico-node-6vxmh                                       1/1     Running            0             19m
calico-system         calico-node-d9t8n                                       0/1     Running            0             17m
calico-system         calico-node-khhpr                                       1/1     Running            0             19m
calico-system         calico-node-nmcds                                       0/1     Running            0             17m
calico-system         calico-typha-d65458ffc-97pn9                            1/1     Running            0             17m
calico-system         calico-typha-d65458ffc-p9cj2                            1/1     Running            0             19m
cattle-fleet-system   fleet-agent-6c857b85b5-zff2l                            1/1     Running            0             17m
cattle-system         cattle-cluster-agent-6f588568-dj7ql                     0/1     CrashLoopBackOff   4 (49s ago)   4m9s
cattle-system         cattle-cluster-agent-6f588568-zl55k                     0/1     CrashLoopBackOff   4 (29s ago)   3m53s
kube-system           cloud-controller-manager-sempre1-ctrl                   1/1     Running            0             20m
kube-system           cloud-controller-manager-sempre1-etcd                   1/1     Running            0             20m
kube-system           etcd-sempre1-etcd                                       1/1     Running            0             19m
kube-system           helm-install-rke2-calico-7dxlb                          0/1     Completed          2             20m
kube-system           helm-install-rke2-calico-crd-wzffm                      0/1     Completed          0             20m
kube-system           helm-install-rke2-coredns-zs9rl                         0/1     Completed          0             20m
kube-system           helm-install-rke2-ingress-nginx-gtkv8                   0/1     CrashLoopBackOff   6 (40s ago)   20m
kube-system           helm-install-rke2-metrics-server-blcf4                  0/1     CrashLoopBackOff   6 (51s ago)   20m
kube-system           kube-apiserver-sempre1-ctrl                             1/1     Running            0             20m
kube-system           kube-controller-manager-sempre1-ctrl                    1/1     Running            0             20m
kube-system           kube-proxy-sempre1-ctrl                                 1/1     Running            0             20m
kube-system           kube-proxy-sempre1-etcd                                 1/1     Running            0             20m
kube-system           kube-proxy-sempre1-wrk1                                 1/1     Running            0             17m
kube-system           kube-proxy-sempre1-wrk2                                 1/1     Running            0             17m
kube-system           kube-scheduler-sempre1-ctrl                             1/1     Running            0             20m
kube-system           rke2-coredns-rke2-coredns-58fd75f64b-kfb69              1/1     Running            0             19m
kube-system           rke2-coredns-rke2-coredns-58fd75f64b-rzpsg              1/1     Running            0             20m
kube-system           rke2-coredns-rke2-coredns-autoscaler-768bfc5985-hcf4b   1/1     Running            0             20m
tigera-operator       tigera-operator-586758ccf7-rc9tq                        1/1     Running            0             19m

Looking the logs seems cluster agent is unable to ping the rancher server, but if I do a curl on the same URL it responds with a pong.
ERROR: <https://rancher75182.senode.dev/ping> is not accessible (Could not resolve host: rancher75182.senode.dev)

helm pods report error as well:
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-ingress-nginx-gtkv8 -n cattle-system
Error from server (NotFound): pods "helm-install-rke2-ingress-nginx-gtkv8" not found
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-metrics-server-blcf4 -n cattle-system
Error from server (NotFound): pods "helm-install-rke2-metrics-server-blcf4" not found

This sounds like an issue with Rancher, not with RKE2?

yes. I can ping the rancher server from all nodes on the downstream cluster

^^ itll look like that

mine only has the nameserver pointing to an ipaddr

Is the name server not reachable from within pods? What do the coredns pod logs say?

coredns pods have the right nameserver and can access from there. Unable to exec omnt the pods which are in a crashloop

So just to confirm, the coredns pods can resolve the rancher server address but the rancher agent pods cannot? How did you test that it can be resolved by coredns?

Recreated a fresh cluster and collected more data. It seems on cluster create the cluster-agent pods successfully come up even though they are unable to ping rancher. Cluster also connects to rancher. One has an error on the certificate and the other cannot reach it. After rebooting the cluster, the cluster-agent pods go in a crashloop

yes, that is the dns service address. Can you show me how specifically you are testing resolution from within the coredns pods?

Just doing a curl as shown above.

You said you were doing that from the local machine or VM, not inside the coredns pod

Both. I also created a new cluster on the same server as Rancher and it seems to work fine. I am able to restart the cluster and it comes back up. I suspect it’s some networking issue on the Harvester server on which the cluster fails to come up

hmm