Hello, I am trying to deploy Neuvector in AWS eks ...
# neuvector_suse-securitym
microscopic-cat-74984
10/30/2025, 9:23 PMHello, I am trying to deploy Neuvector in AWS eks using the neuvector helm chart but when I activate the PVC in the values, the pods come up pending:
controller:nodeSelector:node: basepvc:enabled: truestorageClass: gp3-retaincapacity: 1Gih
hundreds-evening-84071
10/30/2025, 11:50 PMthat is basically what I have but not in eks
Copy code
controller:
enabled: true
pvc:
enabled: true
storageClass: nfs-scm
microscopic-cat-74984
10/31/2025, 2:15 AMYes, gp3-retain is our default storage class. The only really unique thing is it auto sizes the volume. The values don ‘t specify a storage class. If that is empty would that not select the default storage class? I have also tried gp2 which is the original default
s
sparse-carpet-64843
10/31/2025, 9:00 AMThis is probably happening because GP2/GP3 storageclasses on AWS by default only allow RWO PVC mode and NeuVector requires RWX PVC mode for controller pods. You need to provide an storageclass that allow you to create RWX volumes like Longhorn.
m
microscopic-cat-74984
10/31/2025, 1:20 PMThank you for your answer. We have an EFS storage class which is AWS RWX on our cluster provisioned on our cluster so I switched the config for the controller to that storage class put pods are still coming up pending
s
sparse-carpet-64843
10/31/2025, 1:20 PMKubectl describe pod?
m
microscopic-cat-74984
10/31/2025, 2:05 PMI am describing the pvc and if it is pending because AWS is not provisioning the actual volume. So this might be a ticket for AWS.
Normal Provisioning 3m53s (x9 over 8m9s) <http://efs.csi.aws.com|efs.csi.aws.com>_efs-csi-controller-6f5cfb6c69-kr8s6_58bc49d6-c0db-49b3-afdb-e523f24a8595 External provisioner is provisioning volume for claim "neuvector/neuvector-data"Warning ProvisioningFailed 3m53s (x9 over 8m8s) <http://efs.csi.aws.com|efs.csi.aws.com>_efs-csi-controller-6f5cfb6c69-kr8s6_58bc49d6-c0db-49b3-afdb-e523f24a8595 failed to provision volume with StorageClass "efs-sc": rpc error: code = InvalidArgument desc = File System does not exist: Resource was not foundNormal ExternalProvisioning 116s (x26 over 8m9s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner '<http://efs.csi.aws.com|efs.csi.aws.com>' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.✅ 1
microscopic-cat-74984
10/31/2025, 2:17 PMI believe I have found the issue. Although, I have the EFS addon installed (and a role), I do not have a filesystem. We probably removed that when CloudBees CI was removed from our cluster.
s
sparse-carpet-64843
10/31/2025, 2:26 PMGood to hear! Let me know how it went once you fix the issue
h
hundreds-evening-84071
10/31/2025, 2:29 PMif you set controller replica to 1 then you do not need RWX
but I think somewhere in the doc it says for production you want to have more then 1 replica for controller.
s
sparse-carpet-64843
10/31/2025, 2:31 PMthat's exactly the expected behavior
sparse-carpet-64843
10/31/2025, 2:32 PMhowever, I highly recommend to have 3 replicas of the controller to ensure HA
💯 1
m
microscopic-cat-74984
10/31/2025, 5:42 PMI created an EFS file system and then a storage class. Now I am getting a PVC bound to a PV. So that part is solved but now the controller pods are stuck in the init container. Logs show init container is waiting for pods to start. I think still has to do with the volume.
microscopic-cat-74984
10/31/2025, 7:07 PMAny idea any the controller pods are stuck in the init container?
microscopic-cat-74984
10/31/2025, 8:55 PMThere is some thing else I am missing. I tried efs pcs with a test deployment an that did not work either. Thx for your help.
h
hundreds-evening-84071
11/01/2025, 2:06 AMis it something with nodeSelector ?
nodeSelector:node: basem
microscopic-cat-74984
11/01/2025, 2:42 AMOooh, could be. We deploy our base components to a specific node group. I can relax that and see if that frees it up. But the controller is a deloyment and not a deamonSet. Also the test component was a simple nginx deployment in default namespace and no node selector and it did not work either. I think our efs addon might be hose or there is some internal requirement the filesystem needs to have that is covered by our IaC (like specific tags). I created this one manually just to progress with testing. But now that a simple use case fails I can create a ticket with AWS as well. I really appreciate all the answers and support here and will post the solution and some feedback on the chart once I figure this out.
microscopic-cat-74984
11/03/2025, 8:27 PMAfter working with Amazon, we chalked this up with a problem with IAC. So when colleague returns from leave, we will reapply IAC and leave a result here when I have it.
🙌 1
microscopic-cat-74984
11/06/2025, 6:45 PMThe PVC is now working but I getting errors I don’t understand in the controller logs. Are you still willing to help?
2025-11-06T175115.443|ERRO|CTL|orchestration.GetK8sVersion: getVersion fail - code=403, tag=oc, useToken=true
2025-11-06T175115.528|ERRO|CTL|resource.(*kubernetes).RegisterResource: fail to register - error=Failed to discover API group: image.openshift.io(decode error status 404: decode json: invalid character ‘p’ after top-level value) resource=image
2025-11-06T175123.983|ERRO|CTL|utils.ResolveAddrList: cannot resolve - addr=neuvector-svc-controller.neuvector
2025-11-06T175140.523Z [ERROR] agent.server.autopilot: Failed to reconcile current state with the desired state
2025-11-06T175142.144|ERRO|CTL|resource.retrieveSecretData: - err=kubernetes api: Failure 404 secrets “neuvector-store-secret” not found secretName=neuvector-store-secret
2025-11-06T175142.425|ERRO|CTL|kv.restoreRegistry: Failed to read registry directory - fedRole=
2025-11-06T175144.029|ERRO|CTL|kv.clusterHelper.GetObjectCertRev: - cn=rootCA error=Key not found
2025-11-06T175148.012|ERRO|CTL|kv.clusterHelper.GetObjectCertRev: - cn=neuvector-svc-admission-webhook.neuvector.svc error=Key not found
2025-11-06T175153.84 |ERRO|CTL|kv.clusterHelper.GetObjectCertRev: - cn=neuvector-svc-crd-webhook.neuvector.svc error=Key not found
2025-11-06T175212.194|ERRO|CTL|resource.(*kubernetes).RegisterResource: fail to register - error=Failed to discover API group: image.openshift.io(decode error status 404: decode json: invalid character ‘p’ after top-level value) resource=image
2025-11-06T175214.342|ERRO|CTL|kv.clusterHelper.GetObjectCertRev: - cn=neuvector-jwt-signing error=Key not found
2025-11-06T175221.627|ERRO|CTL|system.(*SystemTools).MonitorMemoryPressureEvents: - error=read-only file system
2025-11-06T175221.627|ERRO|CTL|main.main.func4.1: MonitorMemoryPressureEvents - error=read-only file system
2025-11-06T175224.116|ERRO|CTL|rpc.ReportConnections: - error=rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: dial tcp 100.64.8.11618400 connect: connection refused”
2025-11-06T175224.116|ERRO|CTL|rpc.ReportConnections: - error=rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: dial tcp 100.64.8.11618400 connect: connection refused”
2025-11-06T175224.116|ERRO|CTL|rpc.ReportConnections: - error=rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: dial tcp 100.64.8.11618400 connect: connection refused”
2025-11-06T175224.118|ERRO|CTL|rpc.ReportConnections: - error=rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: dial tcp 100.64.74.8118400 connect: connection refused”
2025-11-06T175224.118|ERRO|CTL|rpc.ReportConnections: - error=rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: dial tcp 100.64.74.8118400 connect: connection refused”
s
sparse-carpet-64843
11/06/2025, 10:51 PMHave you ensured all the ports available for NeuVector? https://open-docs.neuvector.com/5.3/special/general#general-guidelines-for-deployment
Copy code
Open required ports on hosts. Make sure the required ports are mapped properly and open on the host. The allinone requires 8443 (if using the console), 18300, 18301, 18400, and 18401. The Enforcer requires 18301 and 18401.m
microscopic-cat-74984
11/07/2025, 2:30 PMIt appears that those errors appeared when recycling the helm install. We have been able to remove many of them by clearing out everything and redeploying. Sometimes all of the pods don’r clean up. We have the leases turned off and the PVC in place. Everything appears operational in the UI. That leaves the errors above about the readonly filesystem, cache sync, and several references to openshift. Here is a sample of the remaining errors. Each controller pod produces slightly different error massages. Only one of the controller pods is issuing connection errors (which leads me to believe that may be a problem). I guess I could try the allinone chart just to make sure everything is open. I am going to move forward with the monitor and circle back. I really do appreciate your responses.
76 Views