https://rancher.com/ logo
Title
a

average-garden-73736

03/30/2023, 5:55 PM
I'm running high load jobs on k3s (embedded etcd, single-node cluster), and I get:
$ kubectl describe nodes

  Warning  ContainerGCFailed  51s (x621 over 10h)  kubelet  rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16782381 vs. 16777216)
And somehow, my node automatically gets placed into a
NotReady
state. Has anyone seen this before?
c

creamy-pencil-82913

03/30/2023, 6:00 PM
Check the logs. ContainerGCFailed suggests that you are low on disk space and it is failing to prune unused images.
👍 1
a

average-garden-73736

03/30/2023, 6:02 PM
Okay, I checked
systemctl status k3s.service
. I realize this issue is not k3s specific. I likely need better garbage collection.
c

creamy-pencil-82913

03/30/2023, 6:24 PM
I would look at the logs not the status
a

average-garden-73736

03/30/2023, 6:29 PM
Logs via
journalctl -u k3s.service
?
1