stats about the nodes, what version of K8S, all that stuff

plz vomit all the details. 😄

So we have 3 clusters that experienced this behavior. We saw it first in our two rancher controllers a day or so after installing NeuVector. We set it to transition modes for new groups for 30 days. No groups have since been transitioned, which makes this extra weird. There are 3924 CVEs detected in this cluster and no vulnerability profiles deployed. I'll summarize this prod cluster later on if it becomes relevant. Rancher Controller Sandbox Sandbox was running 1.33.6 when this happened. Production was running some patch version of 1.32. Rancher in Sandbox was 2.12 and 2.11 in Production. When upgrading our prod environment, we disabled the enforcer pods by setting a deliberately incorrect

nodeName

in the DaemonSet. They're still disabled. This cluster is running on RHEL with Kernel 5.14.0-611.11.1.el9_7.x86_64. Each node has 2 CPUs and 16GB of memory running containerd 2.1.5-k3s1. SELinux is set to enforcing. The Rancher controller is running only Rancher, ArgoCD, and NeuVector. NeuVector is version 5.4.7 and has 44 groups, 108 network rules, 7 disabled response rules, DLP and WAF sensors disabled, admission control is disabled although for some reason the admission control pane doesn't load. The

/admission/state

URL responds with a 404. OIT Sandbox (Downstream) The downstream cluster, has been enforcing for quite some time and only hit that CPU starvation two days ago. It's running on Ubuntu 24.04 with kernel 6.8.0-88-generic. It has 3 control plane/etcd nodes., each with 2 cores and 8GiB of memory. The worker nodes have 8 cores and 32 GiB of memory. We were experiencing memory issues as some of our workloads would run away and trash the node so we added the following kubelet args to all our downstream clusters:

- kube-reserved=memory=1Gi
- system-reserved=memory=2Gi
- eviction-hard=memory.available<1Gi

This cluster has the neuvector-exporter installed, the controller does not. It looks like our enforcers are incorrectly deployed as they're only on the worker nodes. I just noticed that one of the enforcers keeps being terminated with status

255

. The logs of the previous container don't show anything interesting, the last entry is

2025-12-09T18:27:32.662|INFO|AGT|main.(*TaskScanner).scanSecretLoop: SCRT: done - Finished=1 TimeUsed=2.000310409s

. I should be specific, actually, rancher shows the pod having 1 restart "51s ago" but that time never changes. Even if I reload the page it still says "51s ago". The pod view shows the container was terminated on Wed, far more than 51s ago. Importantly, this node is not the one that crashed. It has 102 groups, most are in protect, a few are in monitor and discover. We have a CRD deployed to allow some exceptions for process and network rules. There are 678 rules, of those, 7 are CRD managed. In this cluster, the "Admission Control" pane loads but everything is disabled. WAF and DLP sensors are disabled. NeuVectopr sees 100 containers. Response rules are disabled. There are 6495 detected CVEs. A CRD is accepting several CVEs. General Notes It's important to note that the NeuVector in that Rancher Sandbox cluster is a federated controller to which the downstream cluster federates. NeuVector is seeing 61 containers and 3 nodes (with all roles). Both NeuVectors have default compliant rules enabled and the cis-operator is installed in both clusters. Both NeuVectors have "auto-scan" on. We have 4 more production downstream clusters that have not experienced any issues like this but none of them have configured monitor/protect of any kind.