This message was deleted.

BTW, had running rke2 control plane on this disks and did not see any issues even etcd is more disk hungry

what are you using for kine? the built in sqlite, some external db, or etcd?

@billowy-computer-46613 built in SQLite db. I just ran install script from k3s home page with all defaults

Ok in case of sqlite: i had an issue with 100%cpu usage, and what happened that there were a lot of garbage in the db. Api-server stores every change in etcd as a new revision. If an object is changed the db can grow quite big. Etcd, I mean kine (Kine Is Not Etcd) is doing regularly "compact", which means deleting old revision. In my case the db grow so big (8gb), that even the "compact" command didn't finish, while the next scheduled "compact" wanted to start ... Even a simple

k get po

took 5-6 seconds to complete ... So first of all check how big is your db:

ls -lh /var/lib/rancher/k3s/server/db/state.db

if its in the Gb range, you might have the same issue.

Currently I have pretty small db size - about 50mb

ok, than cleaning isn't needed. But if you have 100% cpu usage, I guess if you run

top

you only see the

k3s

process to consume all cpu. If you use

htop

you can see the htreads, but still wont be able to figure which component is actually eating he cpu. The problem is that k3s by design a single process, instead of the vanilla k8s where you would have: api-server, scheduler, etcd, controller-manager, kubelet, containerd all as a separate process. You have 2 option: you can enable the

pprof

option which is a golang built in profiler: • it will make k3s a little bit slower, but you check various metrics, like what actual code is using the cpu • to do that, you have to install golang on the box, collect metrics, and than use

go tool pprof -top

to find the critical program code An other option that you can extract the "etcd" part out of k3s, by running a

kine --debug

command in /var/lib/rancher/k3s/server/db/, and tell k3s server to connect to that one instead of running it internally in a thread. You have to edit

/etc/systemd/system/k3s.service

and add a

--datastore-endpoint

option