https://rancher.com/ logo
Join Slack
Powered by
Hi everyone, I'm facing a recurring issue with mu...
# general
b
Hi everyone, I'm facing a recurring issue with multiple RKE2 clusters provisioned via Rancher. Every few days, at exactly 00:00 (midnight), the control-plane components freeze or crash, leaving the cluster in an "Updating" state in the Rancher UI. Some days the automatic midnight task runs fine, but other days it completely breaks the cluster. Environment details: Rancher Version: v2.14.1 RKE2 version: v1.34.7+rke2r1 Nodes: 3 control-plane and 9 workers SO Version: Rocky Linux release 9.8 (Blue Onyx) What we've observed & audited so far: ETCD Snapshots: We manually disabled the native ETCD snapshots in the local RKE2 config (etcd-snapshots-disabled: true) thinking it was an I/O conflict, but the midnight crashes keep happening randomly. Logs Behavior: Right after midnight, rke2-server enters a restart loop on the control-plane nodes. The logs show repetitive loops of: "Managed etcd cluster bootstrap already complete and initialized followed by Opening etcd client connection... trying to reach 127.0.0.1:2379." Eventually, the service stops or is killed, leaving orphaned containerd-shim processes with the systemd message: “Processes safely migrated to parent cgroup / remains running after unit stopped”. It seems like an automated Rancher/RKE2 internal task triggering at 00:00 (like certificate rotation checks, or Rancher's own enforced etcd snapshot pipeline overriding the local config) is choking the ETCD/Control-plane. Is there a way to completely reschedule or change the hour of these default midnight Rancher automation tasks? Has anyone experienced similar ETCD quorum drops due to midnight routines, and how did you mitigate them? Any insights or advice on how to debug what specific job runs at 00:00 would be greatly appreciated! Thanks in advance, Denís
Here you can see a log snippet. After this, it enters a restart loop and repeats over and over again:
Jun 13 000034 node2.domain.local rke2[2200124]: time="2026-06-13T000034+02:00" level=error msg="Failed to get recorded learner progress from etcd: context deadline exc>
Jun 13 000104 node2.domain.local rke2[2200124]: time="2026-06-13T000104+02:00" level=error msg="Failed to get recorded learner progress from etcd: context deadline exc>
Jun 13 000106 node2.domain.local rke2[2200124]: time="2026-06-13T000106+02:00" level=fatal msg="leaderelection lost for rke2-etcd"
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 487753 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 487770 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 488524 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 488571 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 488792 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 489201 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 489364 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 491418 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 492225 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 492264 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 492290 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 504936 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 1539925 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 2200766 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 2841861 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 2841894 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 2841948 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 2842200 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Unit process 2842311 (containerd-shim) remains running after unit stopped.
Jun 13 000115 node2.domain.local systemd[1]: rke2-server.service: Consumed 57min 46.210s CPU time, 681.8M memory peak.
Jun 13 000120 node2.domain.local systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 2.
Jun 13 000120 node2.domain.local systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jun 13 000120 node2.domain.local systemd[1]: rke2-server.service: Consumed 57min 46.224s CPU time, 681.8M memory peak.
Jun 13 000120 node2.domain.local systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=warning msg="not running in CIS mode"
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=info msg="Applying Pod Security Admission Configuration"
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=info msg="ETCD snapshots are disabled"
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=info msg="Starting rke2 v1.34.7+rke2r1 (6fb975ad761d191a245a4c0215843e8c19423ac9)"
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Jun 13 000120 node2.domain.local rke2[3242783]: time="2026-06-13T000120+02:00" level=info msg="Opening etcd client connection with endpoints [https://127.0.0.1:2379]"
Jun 13 000130 node2.domain.local rke2[3242783]: {"level":"warn","ts":"2026-06-13T000130.640007+0200","logger":"k3s.remotestore","caller":"v3@v3.6.7-k3s1/retry_intercep>
Jun 13 000130 node2.domain.local rke2[3242783]: time="2026-06-13T000130+02:00" level=fatal msg="Error: preparing server: failed to bootstrap cluster data: failed to re>
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 487753 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 487770 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 488524 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 488571 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 488792 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 489201 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 489364 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 491418 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 492225 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 492264 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 492290 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 504936 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 1539925 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 2200766 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 2841861 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 2841894 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 2841948 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 2842200 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: rke2-server.service: Unit process 2842311 (containerd-shim) remains running after unit stopped.
Jun 13 000130 node2.domain.local systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
b
Is that for all nodes or just node2? It seems to be giving a report on activity then says there's a scheduled restart job with a restart counter. Where might it be getting that from?
"leaderelection lost for rke2-etcd" suggests it's possibly the etcd that's having problems. Does that have anything other than the snapshots that might be getting in the way?
👀 1
b
One or two control-plane nodes end up in this state. It looks like an automatic provisioning or plan execution is triggered every day at that exact time, as you can see in this log snippet: https://pastebin.com/3k2Ps002 We also noticed that the Rancher server itself restarted some pods at midnight (16 hours ago from the time of this check):
kubectl -n cattle-system get po
NAME READY STATUS RESTARTS AGE
rancher-7968b8d96f-7mmdw 1/1 Running 3 (16h ago) 3d7h
rancher-7968b8d96f-gzf78 1/1 Running 3 (16h ago) 3d7h
rancher-7968b8d96f-n2sw7 1/1 Running 1 (40h ago) 3d7h
rancher-webhook-566cbc4db-rhbhb 1/1 Running 0 34d
We suspect this might be due to a disk write performance issue on the etcd nodes.
This is how the cluster management dashboard looks when I log in the morning...

https://dinisinho.gal/wp-content/uploads/2026/06/rancher.png▾

h
was this happening before upgrading to Rocky Linux release 9.8 ?
b
dYes, it also happens on machines running Rocky 9.7.
b
Your dashboard png is not visible. The first line in your pastebin is for "encountered an error while attempting to update the secret: etcdserver". Do you have some regular job for updating the secret? What happens if you change that?
68 Views
Open in Slack
PreviousNext
Powered by