https://rancher.com/ logo
Join Slack
Powered by
With the way that rke2 handles things like deployi...
# rke2
w
With the way that rke2 handles things like deploying ingress controllers via the manifests in the, how would I go about adding additional ingress controllers? i’m struggling to find an example.
c
what do you mean additional ingress controllers? It is possible to run multiple ingress controllers on a single cluster, and select which one you want to use by setting ingressClassName on the ingress resources… but there is not generally any good reason to do that unless you’re trying to migrate from one to another. One controller can support an essentially unlimited number of ingress resources.
w
Well we were planning on having an ingress for each team, something like *.team1.example.com, *.team2.example.com, etc. so each team would have its own ingress. I can’t do a multiple level wild card so i only have an option of a single default cert.
Also, for whatever reason I cannot for the life of me figure out where the “default” ingress controller is defined…in /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml there is “global.systemDefaultIngressClass: ingress-nginx” if i run “kubectl get ingressclass -A” there is a single ingress listed, name = nginx, controller = k8s.io/ingress-nginx if i run “kubectl get ingressclass -o yaml” i would expect to see “ingressclass.kubernetes.io/is-default-class” annotation but i don’t see it.
c
you can have ingress resource per team. You do not need an ingress controller per team. Doing so will make your life unnecessarily complicated.
Just give each team their own ingress with *.team1 *.team2 and so on. 1 controller can manage as many ingress resources, and as many certs, as you need.
what is it that makes you think you need a whole new controller to support multiple ingress resources?
w
Just reading online, maybe I misunderstood what they were meaning. Basically they said “changes on one team shouldn’t impact changes on the other team” Do you happen to have config examples on how that’s done?
c
Just create an Ingress resource for each team, or for each cert/wildcard you need. Point it at whatever backend or service will host it. Its no more complicated than that.
w
hmmm, where is that configured? I guess i’m confusing it in the wrong place… i’m putting the cert in /var/lib/rancher/rke2/server/manifets/rke2-ingress-nginx-config.yaml and in the valuesContent inside of that I specify “extraArgs: defualt-ssl-certificate: “kube-system/mycerts”
c
what?
no. ingress RESOURCES. Not configuration for your ingress controller
Use your ingress resources to configure the certificates for that ingress. Do not configure certificates at the ingress controller level.
You should not need to modify the ingress controller config AT ALL to do this.
You want to use different certificates for different ingress configurations but have somehow decided that the only way to do this is to deploy multiple controllers using different default certificates?
Go read the upstream docs on Ingress resources https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
w
i’m reading up on it. But i’m about 90% sure that this will be app specific
c
what is app specific about this?
Copy code
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: team1
spec:
  tls:
  - hosts:
      - *.<http://team1.example.com|team1.example.com>
    secretName: team1-tls
  rules:
  - host: *.<http://team1.example.com|team1.example.com>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: team1-service
            port:
              number: 80
---
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: team2
spec:
  tls:
  - hosts:
      - *.<http://team2.example.com|team2.example.com>
    secretName: team2-tls
  rules:
  - host: *.<http://team2.example.com|team2.example.com>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: team2-service
            port:
              number: 80
you can provision the tls secrets on your own, or use CertManager to do it for you. easy peasy.
w
hmmm so trying to wrap my head around how this would work for us… what stops a user from using one of the other ingress types? would it be assumed that I have to go and fix/modify each teams ingress like if they needed a different path and port?
I guess I just don’t see how this would work if people are creating their own ingress.
v
yes, what @creamy-pencil-82913 showed you works just fine. You just adjust each ingress resource to the needs of the app as shown in the examples above. In my case, we deploy each app resources (pod, ingress, etc) in its own namespace. Then we group a bunch of namespaces into a Rancher project and give a team access to their project. HTH
w
@victorious-glass-17139 so are y’all doing the deployment for teams then? Like where does your responsibility begin and the app development team begin. I’m not super interested in begin involved in every single minor change and constantly having to futz with “their” ingress.
c
why would you need to do that
this is how ingress is designed to work. many ingress resources, in many namespaces. all managed by a single ingress controller.
w
I mean if they are changing their app to need different services added to the ingress resources wouldn’t they (or an admin) need to change it every time their app changes?
c
this is how the controller pattern works. You don’t deploy one controller for each individual resource. One controller manages all resources of a particular group or kind
this is a policy problem, not a technology problem
do you have some reason that you would not allow them to manage their own ingress resources?
w
Well I was wanting to provide a generic option to them with certs we deploy for a team that we register. And your pattern would work for that, but only pitfall i see is that as an admin i might have to be involved with making changes to their ingress to point to various services
i guess we could just deploy an ingress resource for each namespace?
Sorry, i know it’s probably stupid questions, but just trying to understand best practices to achieve what is needed. Certainly seem to keep getting various methods/answers so trying to wade through it all.
v
@victorious-glass-17139 so are y’all doing the deployment for teams then? Like where does your responsibility begin and the app development team begin.
I’m not super interested in begin involved in every single minor change and constantly having to futz with “their” ingress. (edited)
@witty-computer-79149 My team does the initial deployment of the app (k8s resources and pipeline) and the dev team has access to everything so they can make changes as needed going forward. However, in our case, if everything is working, manual changes after initial deployments are rare. but yeah, our goal is to hand all maintenance to the dev team themselves.
w
@victorious-glass-17139 got it. Yeah, my only concern with this model would be that we are having to to back and forth constantly because they want the ingress changed as the apps are more in development
v
@witty-computer-79149 I see. This is a challenge for us too, we're still working on giving complete access to dev team so they can do all the packaging of their app (via Helm charts) and deployment. Dev team keeps wanting to change the whole app deployment process though 🙂 (mostly cause they're unfamiliar with cloud native tools such as Helm, k8s, ArgoCD, etc)
w
@victorious-glass-17139 yup, very much the same for us, but I’m far from an expert on k8. I have good understanding of “systems” interactions of general systems and their architecture and interactions. Like in this case, I see what nginx-ingress is doing, it’s just proxying traffic, but how things are separated and used across users is kind of hand wavy bc there’s so many options of configuration. Prime example my original question here. there seems to be about N+1 opinions on how to implement it. I got people in k8 slack saying to deploy multi controllers, others is it’s supposed to have single controller and multi resource. I’m still confused on how to make sure teamA doesn’t use teamB.
v
@witty-computer-79149 The way we manage access is via Rancher. We create project A in Rancher and move Teams A's namespaces to project A, and give Team A access to that project. Our Rancher authentication is integrated with Okta (which talks to AD) so users are in AD groups and we have Rancher global roles. However, the role assignment is not straight forward in Rancher, it takes several of us to figure out how to do it and we don't always get it right 🙂 For example I don't think there is an option in Rancher UI to give a group of users rights to a namespace(s), But this may be another topic. To give Team A access to cluster via command line, they login to Rancher (using their Okta account) and download a kubeconfig file from the Rancher UI (top right corner) which should give them access to only namespaces in project they belong to and see in Rancher UI. HTH.
w
@victorious-glass-17139 Yup, that’s understandable, but in this case what if they have multiple namespaces in the project? What project do i put the ingress resource in? We are doing the same thing with user logins except with OIDC.
v
@witty-computer-79149 ok, when I learned k8s I learned the concept of ingress in theory but not in real world, meaning they didn't teach me how to handle multiple apps for multiple teams, etc. They way we handle multiple apps is by creating a namespace for each app where we put all k8s objects including the ingress resource. The only ingress controller we use is the one shipped with RKE2 (nginx-ingress) and its pods run in kube-system namespace. Then we have created a bunch of ingress resources for each app on their own namespaces and this works just fine, no issues there.
w
@victorious-glass-17139 Yeah, that’s the way I understand the recommendation here as well. I was just saying as an example. TeamA gets a group in rancher, then inside of the group there might be teamA-dev, teamaA-test, teamA-prod. So where do you throw the ingress resource? Are you putting an ingress resource in all of the independent namespaces? If you are only making a single ingress for teamA, where do you put? How do you “share” that? Then do you just let the devs go ham and manage the ingress resource at their own leisure?
v
@witty-computer-79149 yes, we create namespaces per app per env, so we have a namespace for dev-myapp, another ns for qa-myapp, and another ns for prd-myapp. Each namespace has its own ingress resource. I know it sounds wacky but that actually works fine for us. And our teams are still in the process of migrating from their previous env to Kubernetes so we're defining policies as we go. Right now my team deploys everything and then hand it to dev team which may manage it via Rancher. Other teams, are still going through us for their changes which again haven't been too much to handle...yet. HTH
w
@victorious-glass-17139 got it so you have an ingress resource per namespace. Makes sense, was just hoping instead of having to create a secret + ingress controller for every single namespace (lots of redundancy) there was a global way to handle it.
👍 1
8 Views
Open in Slack
PreviousNext
Powered by