https://rancher.com/ logo
#k3s
Title
a

adorable-toddler-38962

05/24/2023, 3:57 AM
Hey all I’m having a super weird issue after upgrading my cluster to HA embedded etcd… I still just connect to a single server node for kubectl but it seems pretty often that I’ll get no response or server not responding errors when I do normal operations like listing pods. I’m not sure how to debug this… any ideas? I’ve got a 6 node cluster, 3 of which are server nodes. Centos stream 9 vms on proxmox
c

creamy-pencil-82913

05/24/2023, 4:24 AM
You check the logs for errors?
If they're all vms on the same backing disk it's highly likely your disk doesn't have the iops for your workload + etcd
Etcd writes are highly latency sensitive
a

adorable-toddler-38962

05/24/2023, 5:53 AM
Which logs should I check and what should I look for to diagnose that?
The vms are split across 2 separate machines each running ssds as the backing disks
c

creamy-pencil-82913

05/24/2023, 7:33 AM
Just the k3s journald logs
a

adorable-toddler-38962

05/24/2023, 1:54 PM
So maybe something that would help would be to make my control plane non scheduleable? how do i add the taints to my control plane node to do that? I see this
--node-taint CriticalAddonsOnly=true:NoExecute
under HA external DB, is that the same for embeded etcd?
2 Views