This message was deleted.

but it seemed like awx-operator was complaining about running out of memory, informing kubernetes of it, and kubernetes responded by killing other containers rather than killing the awx-operator container.

that… is not how that works.

Thanks for the reply and helping educate my clown ass. So that's the gap I'm missing in my understanding. If one container hits its memory limit... what should be getting killed by the kernel?

a process in that container

Yea, there's an awx-operator in the awx namespace, and an AWX CRD in the same namespace. The pod has default limits for cpu and memory, cpu is 1500m, memory is 960mi

What exactly do you mean by the RKE2 cluster “dying”

You helped clarify part of it for me because the OOM killed processes on the box seemed to be python3 and ansible-playbook processes. I had assumed those were from AWX job launches, but those are also part of the things running in the awx-operator. Re: dying: my connection to the cluster via k9s terminated several (but not all) times and I noticed certain kube-system processes like canal, kube-proxy being restarted. These coincided with the OOM kills, but it did not occur every time.

I probably would have investigated why those were getting restarted. idk how your AWX deployment is set up, but if it ends up thrashing the disk because it is running out of memory and getting restarted all the time, that can affect the performance of etcd (if they share disk) and then cause greater system instability.

One of the job failures from an AWX job launch did have the error: "receptor detail: error creating pod: etcdserver: request timed out" so that does make sense. The thing about it is that the awx-operator was stuck at 90%+ memory utilization for all day yesterday. It wasn't until I got desperate and out of ideas that I killed it and when it came back the utilization was back to 5% limit utilization. I guess your earlier explanation that it'd just kill pids inside the container solves why it didn't "restart" the container like I'd assume it would. Appreciate it. Thanks for the insights!

You might consider putting your workload or the etcd datastore on a different ebs volume

That's an excellent idea. Is there documentation on how one might be able to do such a migration or would it have to be a new build?

I would do all of /var/lib/rancher/rke2/server/db but yeah

Ok. Thank you very much for helping me! You are a good person.