https://rancher.com/ logo
#general
Title
# general
c

crooked-cat-21365

01/11/2023, 12:28 PM
Trying to setup a new cluster using RKE2 on-premises the control plane doesn't start. The system log shows
Copy code
Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> sh[2138]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> sh[2139]: /bin/sh: 1: /usr/bin/systemctl: not found
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=warning msg="not running in CIS mode"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Starting rke2 v1.24.8+rke2r1 (b061438ce1f04518d980aac842b62f92095d793d)"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Managed etcd cluster initializing"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Starting etcd for new cluster"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Tunnel server egress proxy mode: agent"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,rke2 --authorization-mode=Node,RBAC --bin>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/se>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Server node token is available at /var/lib/rancher/rke2/server/token"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="To join server node to cluster: rke2 server -s <https://172.19.96.128:9345> -t ${SERVER_NODE_TOKEN}"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Agent node token is available at /var/lib/rancher/rke2/server/agent-token"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="To join agent node to cluster: rke2 agent -s <https://172.19.96.128:9345> -t ${AGENT_NODE_TOKEN}"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Run: rke2 kubectl"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="certificate CN=<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> signed by CN=rke2-server-ca@1673437584: notBefore=2023-01-11 11:46:24 +0000 UTC notAfter=2024-01-11 12:02:46 +0000 UTC"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="certificate CN=system:node:<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de>,O=system:nodes signed by CN=rke2-client-ca@1673437584: notBefore=2023-01-11 11:46:24 +0000 UTC notAfter=2024-01-11 12:02:46 +0000 UTC"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Module overlay was already loaded"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Module br_netfilter was already loaded"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 3670016"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Runtime image <http://index.docker.io/rancher/rke2-runtime:v1.24.8-rke2r1|index.docker.io/rancher/rke2-runtime:v1.24.8-rke2r1> bin and charts directories already exist; skipping extract"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Using private registry config file at /etc/rancher/rke2/registries.yaml"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Containerd is now running"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 12.972685ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 1.097175ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 946.039<C2><B5>s"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 1.136069ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 1.041789ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/li>
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Connecting to proxy" url="<wss://127.0.0.1:9345/v1-rke2/connect>"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Handling backend connection request [<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de>]"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:02:48 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2281]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
:
:
224 Views