hello, if one is using rke config:

etcd-snapshot-schedule-cron: '20 */6 * * *'

is there a way to set, that snapshot shall not be done on the leader? ratio: at the time of snapshot, the etcd cluster is unavailable for a short time and some leader elections fail. (e.g., rancher leader election).

Firstly I would investigate the unavailability and try to fix that. As the other option is only bypassing native rke2 functionality by disabling it on each CP node and scheduling snapshots externally. Try increasing election timeout or heartbeat interval, move snapshot out to a dedicated disk and disable compression of the snapshot

not sure, how this could be fixed:

2026-07-11T12:00:35.244208163+02:00 stderr F E0711 10:00:35.242218      43 leaderelection.go:445] "Failed to update lease optimistically, falling back to slow path" err="Put \"<https://1>
0.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers?timeout=15m0s\": context deadline exceeded" lock="kube-system/cattle-controllers"
2026-07-11T12:00:35.24448038+02:00 stderr F E0711 10:00:35.242289      43 leaderelection.go:452] "Error retrieving lease lock" err="context deadline exceeded" lock="kube-system/cattle-controllers"
2026-07-11T12:00:35.244491511+02:00 stderr F I0711 10:00:35.242309      43 leaderelection.go:299] "Failed to renew lease" lock="kube-system/cattle-controllers" err="context deadline exceeded"

it happens exactly in time of etcd doing snapshot. Snap size is about 500MB.

there's just the outcome logged but not the action. I wonder how long did it take before starting snapshot creation until this was logged. Is there a chance this is running on a spinning disk or rather oversaturated storage subsystem? You can increase

etcd-db-metric-poll-interval

and

etcd-compaction-interval

but I wonder if that will have any impact at all. There are also election related defaults you can increase in kube-controller-manager: kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager But I would first confirm there are no frequent leader changes or lease related problems due to disk/network latency, CPU or memory pressure or frequent restarts (evictions, crashes)

it runs from ssd devices only, no rotational. hopefully this is relevant log from the etcd at the same time:

2026-07-11T11:55:51.446420274+02:00 stderr F {"level":"info","ts":"2026-07-11T09:55:51.446247Z","caller":"mvcc/hash.go:157","msg":"storing new hash","hash":2268017395,"revision":1971370137,"compact-revision":1971364818}
2026-07-11T11:59:44.103476757+02:00 stderr F {"level":"info","ts":"2026-07-11T09:59:44.103089Z","caller":"etcdserver/server.go:2217","msg":"triggering snapshot","local-member-id":"ef12a22198b381bf","local-member-applied-index":2183595556,"local-member-snapshot-index":2183585555,"local-member-snapshot-count":10000,"snapshot-forced":false}
2026-07-11T11:59:44.106370649+02:00 stderr F {"level":"info","ts":"2026-07-11T09:59:44.106225Z","caller":"etcdserver/server.go:2262","msg":"saved snapshot to disk","snapshot-index":2183595556}
2026-07-11T12:00:02.911064579+02:00 stderr F {"level":"info","ts":"2026-07-11T10:00:02.910806Z","caller":"fileutil/purge.go:88","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/0000000000000086-00000000822642ce.snap"}
2026-07-11T12:00:42.21567136+02:00 stderr F {"level":"info","ts":"2026-07-11T10:00:42.215454Z","caller":"wal/wal.go:825","msg":"created a new WAL segment","path":"/var/lib/rancher/rke2/server/db/etcd/member/wal/0000000000014b55-00000000822710c3.wal"}
2026-07-11T12:00:50.523398813+02:00 stderr F {"level":"info","ts":"2026-07-11T10:00:50.523128Z","caller":"mvcc/index.go:194","msg":"compact tree index","revision":1971375438}
2026-07-11T12:00:51.534562147+02:00 stderr F {"level":"info","ts":"2026-07-11T10:00:51.534399Z","caller":"mvcc/kvstore_compaction.go:70","msg":"finished scheduled compaction","compact-revision":1971375438,"took":"914.428521ms","hash":325786831,"current-db-size-bytes":506068992,"current-db-size":"506 MB","current-db-size-in-use-bytes":230494208,"current-db-size-in-use":"230 MB"}
2026-07-11T12:00:51.534598906+02:00 stderr F {"level":"info","ts":"2026-07-11T10:00:51.534481Z","caller":"mvcc/hash.go:157","msg":"storing new hash","hash":325786831,"revision":1971375438,"compact-revision":1971370137}
2026-07-11T12:00:59.725137219+02:00 stderr F {"level":"info","ts":"2026-07-11T10:00:59.724944Z","caller":"fileutil/purge.go:88","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/wal/0000000000014b50-000000008225c0a6.wal"}
2026-07-11T12:01:13.57256393+02:00 stderr F {"level":"info","ts":"2026-07-11T10:01:13.572327Z","caller":"etcdserver/server.go:2217","msg":"triggering snapshot","local-member-id":"ef12a22198b381bf","local-member-applied-index":2183605557,"local-member-snapshot-index":2183595556,"local-member-snapshot-count":10000,"snapshot-forced":false}
2026-07-11T12:01:13.576075152+02:00 stderr F {"level":"info","ts":"2026-07-11T10:01:13.575918Z","caller":"etcdserver/server.go:2262","msg":"saved snapshot to disk","snapshot-index":2183605557}
2026-07-11T12:01:32.912669563+02:00 stderr F {"level":"info","ts":"2026-07-11T10:01:32.912432Z","caller":"fileutil/purge.go:88","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/snap/0000000000000086-00000000822669df.snap"}
2026-07-11T12:04:42.641566805+02:00 stderr F {"level":"info","ts":"2026-07-11T10:04:42.641340Z","caller":"wal/wal.go:825","msg":"created a new WAL segment","path":"/var/lib/rancher/rke2/server/db/etcd/member/wal/0000000000014b56-0000000082273fb3.wal"}
2026-07-11T12:04:59.767781681+02:00 stderr F {"level":"info","ts":"2026-07-11T10:04:59.767572Z","caller":"fileutil/purge.go:88","msg":"purged","path":"/var/lib/rancher/rke2/server/db/etcd/member/wal/0000000000014b51-00000000822603f7.wal"}
2026-07-11T12:05:50.529221937+02:00 stderr F {"level":"info","ts":"2026-07-11T10:05:50.528973Z","caller":"mvcc/index.go:194","msg":"compact tree index","revision":1971382954}
2026-07-11T12:05:51.513946107+02:00 stderr F {"level":"info","ts":"2026-07-11T10:05:51.512740Z","caller":"mvcc/kvstore_compaction.go:70","msg":"finished scheduled compaction","compact-revision":1971382954,"took":"940.708031ms","hash":1607211870,"current-db-size-bytes":506068992,"current-db-size":"506 MB","current-db-size-in-use-bytes":259751936,"current-db-size-in-use":"260 MB"}
2026-07-11T12:05:51.514014685+02:00 stderr F {"level":"info","ts":"2026-07-11T10:05:51.512837Z","caller":"mvcc/hash.go:157","msg":"storing new hash","hash":1607211870,"revision":1971382954,"compact-revision":1971375438}

The log looks fine. If you can rule out storage slowness then have a look at kube-apiserver performance and factors that may put too much pressure on it. By storage slowness I mean also slow writes, as sometimes SSD may behave normally and pass the benchmarks but due to condition or cheap controller, writes will suffer.

the machine is dual AMD EPYC 7F52 16-Core Processor, 512GB mem, dual ssds. three such machines running rke2 cluster and hosting only rancher, nothing else there. metrics report maximum etcd sync time is below 5ms. the rancher pod restarts due to leader election are bit repetitive.

that's a beefy setup indeed. I suggest to work with support. they can investigate this in depth with you and find the root cause

will check this, thanks.

taking snapshots should be non-disruptive. I have never heard of this causing any problems for anyone. What specifically are you seeing?

kubectl  get pods -n cattle-system
NAME                              READY   STATUS    RESTARTS        AGE
rancher-ff7f7d8b4-4cftm           2/2     Running   1 (6d14h ago)   8d
rancher-ff7f7d8b4-7zrdv           2/2     Running   1 (5d7h ago)    7d3h
rancher-ff7f7d8b4-fz7h8           2/2     Running   1 (2d8h ago)    8d

all randomly, but the restart time always matches etcd snapshot time on some node

start

iostat -d -x 1

running and then trigger a manual snapshot with

rke2 etcd-snapshot save

. I suspect your disk is struggling more than you expect.

this seems to be maximum burst:

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
md0              0.00      0.00     0.00   0.00    0.00     0.00    4.00     16.00     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
md1              2.00      8.00     0.00   0.00    0.00     4.00   41.00    176.00     0.00   0.00    0.00     4.29    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   9.60
md124            0.00      0.00     0.00   0.00    0.00     0.00  239.00 237056.00     0.00   0.00  224.17   991.87    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   53.58  48.00
md125            0.00      0.00     0.00   0.00    0.00     0.00   58.00    264.00     0.00   0.00    6.28     4.55    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.36  15.20
md126            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
md127            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sda              0.00      0.00     0.00   0.00    0.00     0.00  300.00 237326.50     5.00   1.64   26.01   791.09    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    7.80  46.80
sdb              0.00      0.00     0.00   0.00    0.00     0.00  306.00 240626.50     4.00   1.29   23.79   786.36    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    7.28  46.20
sdc              2.00      8.00     0.00   0.00    0.00     4.00   33.00    157.50     6.00  15.38    0.09     4.77    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdd              0.00      0.00     0.00   0.00    0.00     0.00    8.00     53.50     8.00  50.00    0.12     6.69    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00

and only 2 prints, i.e., all below 3 seconds which should be within all timeouts.

Are you getting slow uo warnings in etcd logs at the same time?

yes, but not terrible, at the time of snapshots, it generates like:

"apply request took too long","took":"162.399707ms","expected-duration":"100ms"

so just slightly above threshold.