https://rancher.com/ logo
Title
l

loud-belgium-84462

08/22/2022, 2:33 PM
Has any one integrated k8s-bigip-ctlr big ip controller with rancher kubernetes
l

little-actor-95014

08/22/2022, 2:55 PM
We tried a few months back and found it to be unreliable and buggy
l

loud-belgium-84462

08/22/2022, 3:00 PM
may i know what kind of issues you ran into. My research says that when it comes to native integration with big ip this controller is our best path. rancher by default installs nginx controller as host port in on prem scenarios. We cannot scale it out and it is not the kubernetes standards
l

little-actor-95014

08/22/2022, 3:06 PM
We ended up creating dedicated ingress nodes that we pointed our BigIP set up at and let K8s do the routing. Looking at my notes our biggest issues were something about HTTPS health probes not being supported and then it's update cycle was very slow. A pod would go down and it would take some time for the controller to then update the F5 with the fact the pod was down. We were also in VXLAN mode, so that may be part of it, but if I recall we asked around and other people weren't very happy with it either. That was some time ago, so things may have changed, but so far we haven't looked back
l

loud-belgium-84462

08/22/2022, 3:14 PM
hmm interesting observations. The issue I have with your set up is that big ip is just proxying to the k8s node in round-robin fashion so no real loadbalancing happens on big ip side. Once it comes to kubernetes in host port fashion then it would route it to the appropriate service. The issue I had with it is as it round robin at times the ingress controller you have would get hot and then clients will be suddenly lost in that set up. We are facing that today, we have ingress controllers that are running in host port fashio
fashion
big ip controller is supposed to do native integration and provide highest value there are 10 m+ downloads apparently. so technically u declare a service of type loadbalancer and it should give u ip from big ip very similar to cloud
the problem with dedicated ingress node is you can only run one ingress on a given node as it is probably a daemon set but even then big ip doesnt do intelligent routing if not 1 lets say is running hot eventually it will cave in and we will lose clients
i am sure you have worker processes running inside the ingress controllers
l

little-actor-95014

08/22/2022, 3:19 PM
Yeah, it's not perfect, but in our case we couldn't accept sending traffic to dead pods and clients getting timeouts over and over. In any case, hopefully you have more success with it than we did
l

loud-belgium-84462

08/22/2022, 3:23 PM
in your case did you suffer clients getting timeouts as well? in our case with current set up we need to restart nginx ingres scontroller from time to time
l

little-actor-95014

08/22/2022, 3:24 PM
We did in our testing, but we also don't use nginx as our ingress
l

loud-belgium-84462

08/22/2022, 3:24 PM
even in your case big ip would randomly send traffic to a kubernetes node meant for ingress and if the ingress is running hot your requests are trapped
oic
i am assuming every ingress controller should have the same issue isnt it? i have tried nginx, traefik same issue under extreme load the ingress gets heated and we tend to loose requests
the problem is big ip is just doing round robbin and hitting random k8s node
just curious what kind of ningress controller u using
l

little-actor-95014

08/22/2022, 3:40 PM
We use Istio
l

loud-belgium-84462

08/22/2022, 3:48 PM
ah ok istio gateway should be better 🙂
so on those dedicated ingress nodes i am guessing you didnt run anything other than istio ingress gateways
i am thinking 6 dedicated nodes for ingress should be a good start
just curious in your env was it rke1 or rke2. rke1 is actually canal which is combo of flannel and calico vxlan. rke2 is calico i believe just curious what was your cni plugin when you tried big ip controller
running into issues 🙂 we have calico on rke1 and f5 controller only works with flannel vxlan in cluster mode. In documentation they do say it supports calico cni but no instructions for it 🙂
l

little-actor-95014

08/24/2022, 9:21 PM
We're RKE2 with Canal. If I recall we did manage to get vxlan working but it wasn't documented. I think we ended up finding the instructions in a recorded lab demo they showed
And you'll always have a "dead" node that's the F5 in the cluster. I think this was the lab https://clouddocs.f5.com/training/community/containers/html/class1/module2/lab1.html
l

loud-belgium-84462

08/24/2022, 9:29 PM
ah thanks so much for sharing the link really appreciate it. we have rke1 and canal i believe is calico + flannel d( for pod ip address). However, canal is not an option for us, we need calico cni for our cluster.