https://rancher.com/ logo
Join Slack
Powered by
Hello all, I recently upgraded k3s from v1.35.5+k3...
# k3s
r
Hello all, I recently upgraded k3s from v1.35.5+k3s1 to v1.36.4+k3s1. Since then, logs are being flooded with (ten) thousands of entries per day stating: 
helpers.go:949] "Eviction manager: no observation found for eviction signal" signal="containerfs.inodesFree"
. Debugging also shows 
cgroupv2 io stats: unknown entry dios=0
hammering the logs. Anyone seen this in v1.36.4?
I have a second node still running v1.35.5+k3s1 (same Ubuntu OS and version) that isn't having these issues. The debug message there is also different: 
level=debug msg="cgroupv2 io stats: skipping over unmappable dios=0 entry"
.
c
Did you enable debug logging or increased verbosity? I suspect you did if you are seeing debug logs...
r
yes I did
you think 
helpers.go:949] "Eviction manager: no observation found for eviction signal" signal="containerfs.inodesFree"
and 
cgroupv2 io stats: unknown entry dios=0
(or the different message in 1.35.5) are not related? I enabled debug mode to figure out why that first message is happening, since it wasn't an issue in the older version.
c
These are all just coming from kubelet or maybe cadvisor. Not anything unique to k3s. Id look there.
r
Thank you, I found the root cause! github.com/kubernetes/kubernetes/issues/139396 this is my "hotfix":
Copy code
$ cat /etc/systemd/system/k3s.service.d/override.conf

# Hotfix <https://github.com/kubernetes/kubernetes/issues/139396>
[Service]
LogFilterPatterns=~.*Eviction manager.*no observation found.*signal="containerfs\.inodesFree".*
🙌 1
8 Views
Open in Slack
PreviousNext
Powered by