stale-painting-80203
03/14/2023, 12:38 AM/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f <https://rancher75182.senode.dev/v3/import/xhctfcnbbt56xvxh6jptq7lzvpw9svd2drkbj5pvm466t5r7zlplqv_c-m-zqcvzlgn.yaml>
<http://clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver|clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver> unchanged
<http://clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master|clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master> unchanged
namespace/cattle-system unchanged
serviceaccount/cattle unchanged
<http://clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding|clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding> unchanged
secret/cattle-credentials-ad9a794 created
<http://clusterrole.rbac.authorization.k8s.io/cattle-admin|clusterrole.rbac.authorization.k8s.io/cattle-admin> unchanged
Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: <http://beta.kubernetes.io/os|beta.kubernetes.io/os> is deprecated since v1.14; use "<http://kubernetes.io/os|kubernetes.io/os>" instead
deployment.apps/cattle-cluster-agent configured
service/cattle-cluster-agent unchanged
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-f75c97ff6-fvb66 1/1 Running 0 19m
calico-system calico-node-6vxmh 1/1 Running 0 19m
calico-system calico-node-d9t8n 0/1 Running 0 17m
calico-system calico-node-khhpr 1/1 Running 0 19m
calico-system calico-node-nmcds 0/1 Running 0 17m
calico-system calico-typha-d65458ffc-97pn9 1/1 Running 0 17m
calico-system calico-typha-d65458ffc-p9cj2 1/1 Running 0 19m
cattle-fleet-system fleet-agent-6c857b85b5-zff2l 1/1 Running 0 17m
cattle-system cattle-cluster-agent-6f588568-dj7ql 0/1 CrashLoopBackOff 4 (49s ago) 4m9s
cattle-system cattle-cluster-agent-6f588568-zl55k 0/1 CrashLoopBackOff 4 (29s ago) 3m53s
kube-system cloud-controller-manager-sempre1-ctrl 1/1 Running 0 20m
kube-system cloud-controller-manager-sempre1-etcd 1/1 Running 0 20m
kube-system etcd-sempre1-etcd 1/1 Running 0 19m
kube-system helm-install-rke2-calico-7dxlb 0/1 Completed 2 20m
kube-system helm-install-rke2-calico-crd-wzffm 0/1 Completed 0 20m
kube-system helm-install-rke2-coredns-zs9rl 0/1 Completed 0 20m
kube-system helm-install-rke2-ingress-nginx-gtkv8 0/1 CrashLoopBackOff 6 (40s ago) 20m
kube-system helm-install-rke2-metrics-server-blcf4 0/1 CrashLoopBackOff 6 (51s ago) 20m
kube-system kube-apiserver-sempre1-ctrl 1/1 Running 0 20m
kube-system kube-controller-manager-sempre1-ctrl 1/1 Running 0 20m
kube-system kube-proxy-sempre1-ctrl 1/1 Running 0 20m
kube-system kube-proxy-sempre1-etcd 1/1 Running 0 20m
kube-system kube-proxy-sempre1-wrk1 1/1 Running 0 17m
kube-system kube-proxy-sempre1-wrk2 1/1 Running 0 17m
kube-system kube-scheduler-sempre1-ctrl 1/1 Running 0 20m
kube-system rke2-coredns-rke2-coredns-58fd75f64b-kfb69 1/1 Running 0 19m
kube-system rke2-coredns-rke2-coredns-58fd75f64b-rzpsg 1/1 Running 0 20m
kube-system rke2-coredns-rke2-coredns-autoscaler-768bfc5985-hcf4b 1/1 Running 0 20m
tigera-operator tigera-operator-586758ccf7-rc9tq 1/1 Running 0 19m
Looking the logs seems cluster agent is unable to ping the rancher server, but if I do a curl on the same URL it responds with a pong.
ERROR: <https://rancher75182.senode.dev/ping> is not accessible (Could not resolve host: rancher75182.senode.dev)
helm pods report error as well:
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-ingress-nginx-gtkv8 -n cattle-system
Error from server (NotFound): pods "helm-install-rke2-ingress-nginx-gtkv8" not found
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-metrics-server-blcf4 -n cattle-system
Error from server (NotFound): pods "helm-install-rke2-metrics-server-blcf4" not found
creamy-pencil-82913
03/14/2023, 12:39 AMthis hostname suggest that you are using a private DNS zone for your Rancher server. Can you confirm that the resolv.conf file on your nodes is properly configured to point at that? Check the RKE2 logs for a message about using 8.8.8.8 instead of your private DNS server.Copy codeERROR: <https://rancher75182.senode.dev/ping> is not accessible (Could not resolve host: rancher75182.senode.dev)
Host resolv.conf includes loopback or multicast nameservers - kubelet will use autogenerated resolv.conf with nameserver 8.8.8.8
stale-painting-80203
03/14/2023, 12:42 AMcreamy-pencil-82913
03/14/2023, 12:42 AMstale-painting-80203
03/14/2023, 12:46 AMcreamy-pencil-82913
03/14/2023, 3:22 AMstale-painting-80203
03/14/2023, 4:03 AMcreamy-pencil-82913
03/14/2023, 4:54 AMstale-painting-80203
03/14/2023, 5:04 AMcurl <https://rancher75182.senode.dev/ping>
pong
creamy-pencil-82913
03/14/2023, 9:01 PMstale-painting-80203
03/14/2023, 9:02 PMcreamy-pencil-82913
03/14/2023, 9:03 PMstale-painting-80203
03/14/2023, 9:05 PMcreamy-pencil-82913
03/14/2023, 9:07 PM