Hi everyone. I installed Neuvector from the App catalogue in Rancher 2.6.5 with defaults (only changed Container Runtime to containerd because I'm using K3s, and changed PVC to use Longhorn), but SSO is not working. Any recommended log that I can review to check what is happening?

Just to check, is your user a rancher admin or cluster owner? Those are the only rancher roles that are allowed in SSO in this version. the

saml-g

INFO message is normal

I'm using the default admin user that is generated when you install Rancher to access Rancher UI. As additional information, I manually set the Rancher's admin password during the installation of Rancher, with the Helm chart, using --set bootstrapPassword flag.

the webui is still not allowing

admin|admin

? that is indeed peculiar 🤔

I am can't reproduce so far on k3s or rke2 using the rancher chart and longhorn. Does it work OK if you don't use a longhorn PVC?

No, even if I not use Longhorn as PVC the error is the same. To add some additional information, my K3S deploy is without Traefik (I'm using NGINX as Ingress) and without servicelb (I'm using kube-vip and MetalLB), and is a HA of 3 nodes beeing Masters and Workers at the same time. K3S is installed on Ubuntu 22.04, with k3sup.

definitely strange. can you check the controller logs when trying to log in, vs the manager logs. and maybe enable debug mode if there is nothing special. Nothing about your arch seems like it would cause an issue. And since the cli works, there is pretty clearly not a connectivity problem between manager and controllers.

Hi. Here are the logs, with debug mode enabled: The Logs from the Manager:

2022-06-28 14:54:39,170|INFO |MANAGER|com.neu.api.AuthenticationService(apply:824): post path auth

2022-06-28 14:54:39,256|WARN |MANAGER|com.neu.api.AuthenticationService(apply:828): Status: 401 Unauthorized

Body: {"code":3,"error":"Authentication failed","message":"Authentication failed"}

2022-06-28 14:54:39,262|INFO |MANAGER|com.neu.api.AuthenticationService(apply:147): saml-g: servername is empty

The Logs from the Leader Controller:

2022-06-28T14:54:38.808|DEBU|CTL|cache.UpdateConnections: - agent=1d1b9febe7e0 app=1001 bytes=592 client=Host:neoappl clientIP=10.12.2.222 clientPort=0 extIP=false external=false first=1656428055 host=neoappliance-k8s-02:6dce4d56-1a13-a2b1-74e8-b16dd06f283e ingress=true ipproto=6 last=1656428055 local=true network= policyAction=1 policyID=10005 policyViolates=0 scope=global server=793ca6076462 serverIP=192.168.193.66 serverPort=80 sessions=0 threatID=0 threatSev=0 toSidecar=false xff=false

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexLock: Acquire ... - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexUnlock: Released - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.UpdateConnections: - agent=1d1b9febe7e0 app=0 bytes=9760 client=c16edcf22e89 clientIP=192.168.193.125 clientPort=0 extIP=false external=false first=1656428072 host=neoappliance-k8s-02:6dce4d56-1a13-a2b1-74e8-b16dd06f283e ingress=false ipproto=6 last=1656428072 local=false network= policyAction=0 policyID=0 policyViolates=0 scope=global server=nv.ip.kubern serverIP=10.43.0.1 serverPort=443 sessions=0 threatID=0 threatSev=0 toSidecar=false xff=false

2022-06-28T14:54:38.808|DEBU|CTL|cache.graphMutexUnlock: Released - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=318

2022-06-28T14:54:38.808|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=318

2022-06-28T14:54:42.706|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=1100956

2022-06-28T14:54:42.706|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=1100956

2022-06-28T14:54:42.707|DEBU|CTL|cache.graphMutexLock: Acquire ... - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexLock: Acquire ... - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexUnlock: Released - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.preProcessConnectPAI: Ignore ingress connection from nv device - client=192.168.67.168 server=192.168.164.101

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexRLock: Acquire ... - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexRUnlock: Released - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexLock: Acquire ... - goroutine=318

2022-06-28T14:54:42.707|DEBU|CTL|cache.cacheMutexUnlock: Released - goroutine=318

do you see anything like this in your controller debug, this is showing a failed login. I dont even see the login attempt (first line) in what you pasted above, as if the auth connection does not even arrive.

Can you hit the NeuVector UI directly at

https://<the.ui.service.ip>:8443

?

@ripe-actor-83292, the NeuVector UI internally is HTTPS or HTTP on port 8443?

HTTPS

Ok, let me try with a kubectl port-forward

thanks for playing along. 🙂 This is definitely a weird one.

No @ripe-actor-83292, the "prod" word here is maybe oversized. Basically refer to the idea that Flannel is not an option in the future for a prod environment as our network engine. We are still testing what we are going to use for Containers Workloads, and clearly Rancher/K3S/Neuvector is an option, but we still don't know for sure.

I'm looking into behavior when rancher auth provider is enabled. It could be the culprit if cli is OK (I also verified cli and ui should use the same api endpoints for authentication)

Curious, if you open your browser dev tools and try to log in, do you see any console errors? Or anything interesting in the returned responses?

Curious, can you post your values.yaml please?

@ripe-actor-83292, I cannot access to the UI even doin a kubectl port-forward, I get a Connection Reset whe I try to access from the browser:

root@appliance-k8s-01:/home/aguida# kubectl port-forward service/neuvector-service-webui -n cattle-neuvector-system 8443:8443

Forwarding from 127.0.0.1:8443 -> 8443

Forwarding from [::1]:8443 -> 8443

Handling connection for 8443

Handling connection for 8443

E0629 18:36:17.344121 2335862 portforward.go:406] an error occurred forwarding 8443 -> 8443: error forwarding port 8443 to pod 4d0d9e20c0c56b4f6a26e9e3ebc997be715f11798662f801b60f6069900dadfb, uid : failed to execute portforward in network namespace "/var/run/netns/cni-e8f9fd3b-9834-d83b-0260-a767438f30e6": read tcp4 127.0.0.1:54932->127.0.0.1:8443: read: connection reset by peer

E0629 18:36:17.344927 2335862 portforward.go:234] lost connection to pod

This is due to a bug in kubectl 1.23+ . if you have a 1.22 kubectl, it should work

@quiet-fountain-46593 I need to have Kubernetes 1.22 too, or only replacing the kubectl binary with 1.22 version should it work?

just using the different kubectl should work

@ripe-actor-83292, can't login with port-forward, with default admin/admin credentials:

ok. check some of the other replies/suggestions here

will try and reproduce myself with an unreachable RANCHER_EP as well, and see if the behavior is the same

@quiet-fountain-46593, RANCHER_EP inside neuvector-controller resolve to the https://rancherFQDN, where rancherFQDN is the FQDN that I configured to access Rancher when I installed it with the Helm Chart. The resolution of that FQDN is only possible from my machine, where I added a line with "IP FQDN" to my local hosts file. Running nslookup inside neuvector-controller, It cannot resolve the FQDN. What I don't understand, understanding that maybe that is the problem that is not allowing me to SSO to Neuvector from Rancher, is why I cannot login to NeuVector with default admin/admin credentials doing the kubectl port-forward, but I can login to Neuvector-Controller from Neuvector-Manager pod using the CLI, with default credentials. Weird, 😑

yea, this could be a poor failure

Ok @quiet-fountain-46593 , I'm going to try. Thanks all for your help.

Going all the way back to your first post - “k3s” should be your selected runtime, not containerd. And what CNI are you using if not Flannel?

I'm using Calico @adventurous-battery-36116

Calico. …which will present some issues with NeuVector until some time later this year

I'm selecting K3S as runtime when I install NeuVector @adventurous-battery-36116, was a mistake when I wrote that message

ok cool - just checking 🙂

@ripe-actor-83292 which is the recommended CNI to use with NeuVector, the most "compatible" one?

UGH! what I said was a typo; sorry. It’s Cilium that has that limitation. Calico is fine

What (if any) network policies are in place?

kubectl get NetworkPolicy -A

Network Policies @adventurous-battery-36116:

root@appliance-k8s-01:/home/aguida# kubectl get NetworkPolicy -A
NAMESPACE                   NAME                POD-SELECTOR     AGE
calico-apiserver            allow-apiserver     apiserver=true   13d
cattle-fleet-local-system   default-allow-all   <none>           13d

if you have CLI access, run

show user

Hi @clean-magazine-25026. On one of my previous posts, I uploaded an image that show the admin user and a user that I created for me, aguida, from the CLI. So the authentication from the CLI os working right.