This message was deleted.
# rke2a
adamant-kite-43734
07/11/2023, 1:16 AMThis message was deleted.
b
blue-kitchen-51801
07/11/2023, 11:49 AMdo you have static allocated IP addresses for each node?
blue-kitchen-51801
07/11/2023, 11:50 AMusually, when the IP is changed, etcd has the old IP and will try to connect to it.. and when etcd is down, kubectl is down also
s
stale-painting-80203
07/11/2023, 12:49 PMAll IPs are static and have not changed.
👍 1
b
blue-kitchen-51801
07/11/2023, 12:51 PMit would be useful to have more logs..
for eg.
journalctl -u rke2-server.service --no-pager -n 1000s
stale-painting-80203
07/11/2023, 1:10 PMattached the journalctl logs
b
blue-kitchen-51801
07/11/2023, 1:12 PMi would suggest restarting the node and then collect journalctl logs again
blue-kitchen-51801
07/11/2023, 1:13 PMPS. I’m not part of Rancher team or an expert in RKE2, i’m just trying to help with the experience I have while running our own RKE2 clusters
s
stale-painting-80203
07/11/2023, 1:17 PMlogs after restarting node
stale-painting-80203
07/11/2023, 1:20 PMI have three nodes: 172.16.1.11, 172.16.1.12, and 172.16.1.13. I did try restarting all three yesterday, but it did not resolve the issue
b
blue-kitchen-51801
07/11/2023, 1:23 PMcould you please dump the journalctl again? i’m not able to find any errors (
"level":"error"blue-kitchen-51801
07/11/2023, 1:23 PMWaiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Errorblue-kitchen-51801
07/11/2023, 1:29 PMI would debug why etcd is not up and running
<etcd-endpoints://0xc002dbe700/127.0.0.1:2379> transport: authentication handshake failed: context deadline exceededblue-kitchen-51801
07/11/2023, 1:31 PMalso, i would look also in the other log files described in https://gist.github.com/superseb/3b78f47989e0dbc1295486c186e944bf#logging
s
stale-painting-80203
07/11/2023, 1:35 PMSince I am unable to run kubectl commands, this may not be easy.
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes👍 1
stale-painting-80203
07/11/2023, 1:36 PMThanks for the suggestions though.
b
blue-kitchen-51801
07/11/2023, 1:37 PMyou might also try these commands (https://gist.github.com/superseb/3b78f47989e0dbc1295486c186e944bf#on-the-etcd-host-itself)
Copy code
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
etcdcontainer=$(/var/lib/rancher/rke2/bin/crictl ps --label io.kubernetes.container.name=etcd --quiet)
/var/lib/rancher/rke2/bin/crictl exec $etcdcontainer sh -c "ETCDCTL_ENDPOINTS='<https://127.0.0.1:2379>' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table"s
stale-painting-80203
07/12/2023, 10:42 PMFigured out what caused the failure and resolved it for now. Seems installing docker on the host machine breaks RKE2 networking somehow. Filed a bug: https://github.com/rancher/rke2/issues/4472
👍 1
1871 Views