Hi folks, migrating the kvm to new hardware I lost...
# rke2
c
Hi folks, migrating the kvm to new hardware I lost a control plane node ("rancher01a"). rancher01b and rancher01c are fine. Apparently the rke2-server doesn't work anymore. The log file says
Copy code
:
Oct 03 13:06:07 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:07+02:00" level=info msg="Starting etcd for existing cluster member"
Oct 03 13:06:07 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:07+02:00" level=info msg=start
Oct 03 13:06:07 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:07+02:00" level=info msg="schedule, now=2023-10-03T13:06:07+02:00, entry=1, next=2023-10-04T00:00:00+02:00"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client->
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=>
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/rke2/server/etc/cloud-config.yaml>
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Server node token is available at /var/lib/rancher/rke2/server/token"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="To join server node to cluster: rke2 server -s <https://172.19.96.170:9345> -t ${SERVER_NODE_TOKEN}"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Agent node token is available at /var/lib/rancher/rke2/server/agent-token"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="To join agent node to cluster: rke2 agent -s <https://172.19.96.170:9345> -t ${AGENT_NODE_TOKEN}"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Run: rke2 kubectl"
Oct 03 13:06:08 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:08+02:00" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
Oct 03 13:06:09 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:09+02:00" level=info msg="Password verified locally for node rancher01a.ac.aixigo.de"
Oct 03 13:06:09 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:09+02:00" level=info msg="certificate CN=rancher01a.ac.aixigo.de signed by CN=rke2-server-ca@1683643043: notBefore=2023-05-09 14:37:23 +0000 UTC notAfter=2024-10-02 11:06:09 +0000 UTC"
Oct 03 13:06:09 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:09+02:00" level=info msg="certificate CN=system:node:rancher01a.ac.aixigo.de,O=system:nodes signed by CN=rke2-client-ca@1683643043: notBefore=2023-05-09 14:37:23 +0000 UTC notAfter=2024-10-02 11:06:09 +0000 UTC"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Module overlay was already loaded"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Module br_netfilter was already loaded"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Runtime image index.docker.io/rancher/rke2-runtime:v1.26.7-rke2r1 bin and charts directories already exist; skipping extract"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml to set cluster configuration values"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"
Oct 03 13:06:10 rancher01a.ac.aixigo.de rke2[901]: time="2023-10-03T13:06:10+02:00" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd"