https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
Writes to etcd need to be written and fsync’d to disk on 3/5 (quorum) of nodes before the write is acknowledged. If there are other writes to the disk on the shared nodes, the etcd fsync will need to flush all that out as well. That is probably where the latency is coming from.
You might try removing the 2 shared-disk nodes from the cluster, making them agents or something - and see if you get fewer errors.
or alternatively, get dedicated disk for etcd on those nodes as well.
n
no matter that fsync metrics look ok?
c
averages can hide things
you are getting latency warnings, so clearly it is not ok
again, averages
but even there it is obvious to me which 2 nodes are on shared disk
8 and 9 right?
n
yes
c
you should probably be looking at p95 or p99 to see the spikes. not the averages
on average it is OK. but the spikes are what cause warnings
n
those charts are quantiles, not avg I guess: 
histogram_quantile(0.95, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[$__rate_interval])) by (instance, le))
c
ok
there’s no legend in your screenshot so
n
also, the etcd perf seems to be a bit slow (slowest request), but it also shows very low variance:
Copy code
PASS: Throughput is 151 writes/s
PASS: Slowest request took 0.067341s
PASS: Stddev is 0.001836s
so I would guess this should be seen even in avg
I can definitely try to remove the two slow nodes tomorrow. Just running a bit out of ideas where the problem could be and where to look
noticed another message: 
026-03-17T19:35:36.262841758+01:00 stderr F {"level":"warn","ts":"2026-03-17T18:35:36.262458Z","caller":"rafthttp/stream.go:529","msg":"dropped internal Raft message since receiving buffer is full (overloaded network)","message-type":"MsgHeartbeatResp","local-member-id":"e711b481db02f5dc","from":"4d2a07f635c6bbc2","remote-peer-id":"e711b481db02f5dc","remote-peer-active":true}
c
That could be resources perhaps. How is network between nodes? Are the etcd pods being throttled?
n
the issue here seems to be the dedicated nvme disk actually, as it complains about delays: 
<http://kub-a6.priv.cerit-sc.cz|kub-a6.priv.cerit-sc.cz> kernel: nvme nvme0: I/O tag 577 (5241) QID 2 timeout, completion polled
and I just witnesses that the delay is for more than 7 seconds, most likely about 32 seconds.
c
oof
overheating perhaps?
n
or maybe firmware issues, not uncommon unfortunately. I moved all etcd instances to shared disks and things settled down.
✅ 1
c
interesting, would not have guessed!
n
btw, I noticed that rke keeps several versions of helm secrets for install jobs and e.g., calico crds are about 1.3MB large, it also contribute to slowing downs as rancher tries to fetch all secrets sometimes. is there a way to limit rke components to keep only one helm revision?
c
A couple extra secrets shouldn't make a noticeable difference... you can use the helm CLI to clean up old releases if you want.
Helm keeps 10 old release revisions by default, and we don't override that.
n
ok, thanks.
just thought if I can set it in Helm resource for rke components, but definitely can remove it manually
all secrets dumped into json take about 300MB ;)
18 Views
Open in Slack
PreviousNext
Powered by