This message was deleted.
# k3s
a
This message was deleted.
d
try to install some monitoring tool or a simple htop inside to see what is running inside
try to set some limits and requests in your services if there are not configured
thats to diagnose what is causing that
maybe there is a OOM Kill in some place
c
Thanks for the suggestions! I am running kube-prometheus-stack and seeing a high RPC rate on etcd (~300 ops/s), and high work queue latency (10s) for some parts of the kubernetes API. Are you suggesting running htop on the host OS? If so, I'm only seeing the high CPU being attributed to the k3s service and no breakdown beyond that. I also have the below api limits on my `config.yaml`:
Copy code
kube-apiserver-arg:
 - 'max-requests-inflight=100'
 - 'max-mutating-requests-inflight=50'
kube-controller-manager-arg:
 - 'kube-api-qps=20'
 - 'kube-api-burst=30'
d
depending on what you find try to add some swap if possible
c
you didn’t say what version you’re on, but this sounds a lot like https://github.com/k3s-io/k3s/issues/12127
👀 1
c
Ha! I just came back to this chat after digging that up as well! I just started using Harbor as a pull-through image registry anyway, so I am going to try disabling Spegel registry mirroring to see if that clears things up. Thanks!
d
I contribute into Harbor @clean-piano-40966 in case if there is something need it
c
@damp-xylophone-94549 Perfect, I'll keep that in mind thank you! @creamy-pencil-82913 I'm on
v1.31.7+k3s1
and it's (currently) looking like turning off that image registry mirroring has reduced my cpu. Love to see graphs like this:
c
it’s fixed for this month’s releases, if you do eventually want to turn it back on again
c
Awesome, thank you for the insight.