https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
It’s set to LoadBalancer by default, but it’s disabled, since rke2-ingress-nginx is deployed as a daemonset. Did you want to change that and/or enable the service?
note that the service is not called 
nginx
, it will be rke2-ingress-nginx-xxxx, and will not exist by default, are you sure that you’re looking at the right service?
c
No probably i'm not looking at the right service
I didn't know the service was disabled by default
How do I enable the nginx service? the goal is to publish a rancher ui 🙂
c
you don’t need to do that.
it’s a daemonset by default. the ingress is available on all the rke2 nodes on ports 80 and 443.
So, just direct your traffic to one of the nodes.
c
at the moment I get that my website is not secure
so i thought i was doing something wrong
and re considered the infra that's why i'm asking
c
RKE2 does not include a LoadBalancer controller by default, if you enabled the service and set it to LoadBalancer you would need to also deploy a cloud provider, or kube-vip, or metallb, or something else to provide a loadbalancer for you.
c
yes currently i'm deploying kube-vip for the HA of rke2
I have correctly a VIP attached to my cluster
c
ok. In that case it should work to switch it over to a LB service. But I don’t think that’s the problem.
c
the only issue i have is the "bad certificate"
everything works
c
It sounds like your ingress certs aren’t set up right? or you are trying to access the nodes by IP, and the ingress is expecting a hostname?
c
Copy code
cattle-system   rancher   <none>   mpro.rancher.<edit>.io   192.168.192.170,192.168.192.49,192.168.194.99   80, 443   3h53m
this is the ingress
everything is managed by rancher itself
and everything works properly, just the ui cert isn't alright, i have cert-manager working and issuing the certificate properly
Copy code
apiVersion: v1
items:
- apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
  kind: Ingress
  metadata:
    annotations:
      <http://cert-manager.io/issuer|cert-manager.io/issuer>: rancher
      <http://cert-manager.io/issuer-kind|cert-manager.io/issuer-kind>: Issuer
      <http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>: '[{"addresses":["192.168.192.170","192.168.192.49","192.168.194.99"],"port":443,"protocol":"HTTPS","serviceName":"cattle-system:rancher","ingressName":"cattle-system:rancher","hostname":"<http://mpro.rancher.k8s.appscaler.io|mpro.rancher.k8s.appscaler.io>","path":"/","allNodes":false}]'
      <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: rancher
      <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: cattle-system
      <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "30"
      <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "1800"
      <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "1800"
    creationTimestamp: "2024-01-26T15:55:12Z"
    generation: 1
    labels:
      app: rancher
      <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
      chart: rancher-2.7.9
      heritage: Helm
      release: rancher
    name: rancher
    namespace: cattle-system
    resourceVersion: "63167"
    uid: 312a3d07-23c4-4ae2-9141-d74b32bef3e5
  spec:
    rules:
    - host: <http://mpro.rancher.io|mpro.rancher.io>
      http:
        paths:
        - backend:
            service:
              name: rancher
              port:
                number: 80
          path: /
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - <http://mpro.rancher.io|mpro.rancher.io>
      secretName: tls-rancher-ingress
  status:
    loadBalancer:
      ingress:
      - ip: 192.168.192.170
      - ip: 192.168.192.49
      - ip: 192.168.194.99
kind: List
metadata:
  resourceVersion: ""
And here's the ingress yaml..
Everything is handled by rancher 🤔
c
right, so does 
<http://mpro.rancher.io|mpro.rancher.io>
point at 192.168.192.170,192.168.192.49,192.168.194.99 in dns?
also, I suspect you don’t own rancher.io… so that’s an interesting choice of hostname for your rancher instance…
c
no i just wanted to edit out
the other part sorry XD
c
ah ok. Well whatever your actual hostname is on that ingress, needs to point at those IPs.
c
one sec, let me explain and pardon my newbieness
on aws route53 i have one public IP that is nat 1:1 to the vip that handles 192.168.192.170,192.168.192.49,192.168.194.99
c
the ingress is set to match on
Copy code
host: <http://mpro.rancher.io|mpro.rancher.io>
or
Copy code
tls:
    - hosts:
      - <http://mpro.rancher.io|mpro.rancher.io>
so if you don’t attempt to connect to it at that hostname, you will get the ingress default certificate. instead of the certificate you configured for rancher.
that hostname needs to point at those hosts, on port 80 and 443
or optionally a VIP in front of those IPs that will get traffic there.
without offloading TLS or HTTP
c
the thing is, by running helm install rancher ... with options, it created automatically the ingress with those ip addresses
those 3 ip addresses
c
rancher didn’t do that. the ingress controller did that.
the ingress controller knows where it’s running, and puts its IPs into the ingress status
it doesn’t know anything about your route53 or the vip. it just knows where it is.
c
I don't understand, my networking is really bad uff
inside the ingress instead of those 3 private ip that are my the ip of each of the node composing the rke2 cluster what should be there?
c
nothing, that is what should be there when it is running as a daemonset.
It all looks to me like it is set up properly.
When you were testing it, what exactly were you hitting? Were you using the hostname, or the vip address, or one of the node IPs?
c
the rancher page is still online
i was trying to reach it via mpro.rancher.io
c
and what is wrong with it?
c
i can reach it, just it says bad cert
c
Does the 
tls-rancher-ingress
secret contain a valid cert that is trusted by your browser?
When you installed the rancher chart, how did you configure for the certificate?
c
I kept the default helm setting
so it should be rancher managed
c
right but we can’t magically acquire a trusted cert for you
c
i set up cert-manager
c
is this exposed to the internet, where the cert-manager challenges will actually work?
c
and yes the tls-rancher-ingress secret contains a valid cert split in the usual 3 keys
c
If you’re using cert-manager, it needs to be able to complete one of the validation challenges.
c
cert-manager, isn't exposed to the internet...
c
welp
if your http endpoint isn’t accessible to the internet then you can’t use letsencrypt http challenges
letsencrypt will work with the ingress to complete the challenge
if the endpoint is accessible to the internet
if its not, then you need to bring your own cert
c
but mpro.rancher.io is accessible by 0.0.0.0/0 on port 443
if you mean that
c
you don’t actually own that domain
c
can i dm you
the domain
c
ah you mean that your actual domain is exposed to the internet?
c
yes
you technically could reach it if i gave you the right url
c
OK. I would check the cert-manager certificate status to see if it completed the challenge. If it did then you should see a cert-manager provisioned cert in that secret.
If the cert in the secret does not match the cert your browser is getting, then something else is going on. Maybe the VIP is pointing to the wrong ports, maybe you have something else terminating SSL in front of the ingress.
OK so that looks good! Is the cert in that secret the same cert that your browser doesn’t like?
c
How do i check that pardon my ignorance?
c
look at the cert in the browser’s site info. look at the cert in the secret (you’ll probably need to base64decode it and feed it into openssl to dump the info)
c
I need to decode all 3 ca.crt tls.crt and tls.key
c
no just tls.crt
c
done, the ssl decoder matches the data on the mpro rancher ui
let me try to give a little more background. I'm on openstack: • 3 nodes with rke2 • 1 private vip that is handled by kube-vip • 1 public ip address associated to the private vip
c
ok. so you have a valid letsencrypt cert, but your browser doesn’t like it. What doesn’t it like about it? doesn’t trust the issuer? hostname doesn’t match the cert? something else?
I am assuming that the cert is actually from LE and not just the default ingress cert, since the cert-manager status indicated that it was ready.
why isn’t it valid
c
ssl shopper says this.. The certificate is not trusted in all web browsers. You may need to install an Intermediate/chain certificate to link it to a trusted root certificate. Learn more about this error. The fastest way to fix this problem is to contact your SSL provider.
a
Does it complain if you close the browser and try again?
c
Hi simon
Yes if i try in incognito it throws the same error
c
that looks like you’re hitting the rancher pod, not the ingress
Did you do something to expose rancher directly, without going through the ingress?
a
Doesn’t look like letsencrypt?
c
no. that is the fake cert signed by the rancher pod’s dynamiclistener CA. it is not a letsencrypt cert.
Is that the same cert you see in the secret?
if it was from LE you would see LetsEncrypt listed in the issuer info, not dynamiclistener
c
So to answer your first question, no i didn't do anything beside following step by step the docs from rancher
yes the dynamiclistener CA is what I get if I decode the .tls part of the certificate in ssl shopper
Copy code
-----BEGIN CERTIFICATE-----
MIICfTCCAiSgAwIBAgIRAIqgSzPxbCpOCeSYfg0ZfNkwCgYIKoZIzj0EAwIwRjEc
MBoGA1UEChMTZHluYW1pY2xpc3RlbmVyLW9yZzEmMCQGA1UEAwwdZHluYW1pY2xp
c3RlbmVyLWNhQDE3MDYyODQ1NjMwHhcNMjQwMTI2MTU1NjE3WhcNMjQwNDI1MTU1
NjE3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2XZzc6n0DiZ5
zw/QR8Wzby11URC8Ylxj5yr21C7njUUgkcJbcLKg9WhSpoPZNX6dfmgn6KFqA/Vz
vdAQicFN0TAmb0y+zLdHljNJNOal91cT1pIkEb8UBgxxI1vtugj4yw9vWM658DK6
rPZ6LhVO5t6dxvy6MYDX27SWTAukyqrTy2IFYz3hf1ZS433PSMHZNMcR9N+Asko4
hYXoWMki7Td1d47hujoYuOWhZ4D586IEBX2Pxmejozt9op1ks5E02jkptZnqxrDI
sCqN+MJdNWKK8V8PY1jn/4YOUMkhUIvTbLpPV30vCdcackmdrd2NTA/7yuPSXMpj
UDsMirq0UwIDAQABo24wbDAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAf
BgNVHSMEGDAWgBQhIhAaxTgNDjETStdzZ6cPoIpeIDArBgNVHREBAf8EITAfgh1t
cHJvLnJhbmNoZXIuazhzLmFwcHNjYWxlci5pbzAKBggqhkjOPQQDAgNHADBEAiAK
QNnQ8nhYiUuuPsybD77cKVKheZikv8UnZhsjH0/MfgIgK43EIQKhckrh0vusA8/x
yG+K6TYnF5jYQ+5wNjFVyCA=
-----END CERTIFICATE-----
c
Did you set the tls source to letsEncrypt?
by default it is set to 
rancher
which means you will need to log in to rancher once it is up and manually provide a valid cert
c
No i left it to rancher
c
if you want to use LE you need to set the tls source and letsencrypt email correctly
c
Do i really have to provide a cert manually if i left it to rancher
c
you could try just updating the chart to change those values?
c
No i didn't mean it in an aggressive way sorry if it sounded like that
c
If you were following the docs, this is noted at the top of the page: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/other-insta[…]-methods/rancher-behind-an-http-proxy/install-rancher
For this tutorial you are going to automatically issue a self-signed certificate through cert-manager. In a real-world use-case you will likely use Let’s Encrypt or provide your own certificate.
c
It was because a colleague of mine provisioned another cluster similarly and I think he did it via tls source set to rancher and his url doesn't have issues with a certificate
c
he would have had to give it a real cert somehow
you need to either give it a cert, or enable LE so it can go out and get one.
c
let me check on his cluster
if there's LE deployed on it
omg
hostname: <url> ingress: tls: source: letsEncrypt letsEncrypt: email: <email> ingress: class: nginx
so he used let'sEncrypt
c
haha yeah so he probably set those two values when installing the rancher chart
c
like i'm going crazy
for the past 4-5 hours thinking about this
Just to give it a shot
I would need to redeploy the whole rancher?
a
helm upgrade?
c
Copy code
helm upgrade rancher rancher-stable/rancher \
  --namespace cattle-system \
  --set hostname=<myhostname> \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=<email> \
  --set letsEncrypt.ingress.class=nginx
i just removed the bootstrap password
do you think the above command will work?
a
try and see 🙂
c
it rolled out correctly
but the certificate in the url didn't change
even if i try incognito mode
a
it might not be immediate, but I think in cattle-system there is tls-ingress secret
which has tls.crt, tls.key
c
tls-rancher-ingress
this one
a
yes
c
it is still the same one from 5h8m ago
a
that date doesn't tell everything 😄
only creation even if it changes in between
well if it re-created then yes
c
so
i mean
i thought it would be re-created since a new cert issued by LE should be there
a
Can you see what is set now?
c
yes it's this one
Copy code
-----BEGIN CERTIFICATE-----
MIICfTCCAiSgAwIBAgIRAIqgSzPxbCpOCeSYfg0ZfNkwCgYIKoZIzj0EAwIwRjEc
MBoGA1UEChMTZHluYW1pY2xpc3RlbmVyLW9yZzEmMCQGA1UEAwwdZHluYW1pY2xp
c3RlbmVyLWNhQDE3MDYyODQ1NjMwHhcNMjQwMTI2MTU1NjE3WhcNMjQwNDI1MTU1
NjE3WjAAMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2XZzc6n0DiZ5
zw/QR8Wzby11URC8Ylxj5yr21C7njUUgkcJbcLKg9WhSpoPZNX6dfmgn6KFqA/Vz
vdAQicFN0TAmb0y+zLdHljNJNOal91cT1pIkEb8UBgxxI1vtugj4yw9vWM658DK6
rPZ6LhVO5t6dxvy6MYDX27SWTAukyqrTy2IFYz3hf1ZS433PSMHZNMcR9N+Asko4
hYXoWMki7Td1d47hujoYuOWhZ4D586IEBX2Pxmejozt9op1ks5E02jkptZnqxrDI
sCqN+MJdNWKK8V8PY1jn/4YOUMkhUIvTbLpPV30vCdcackmdrd2NTA/7yuPSXMpj
UDsMirq0UwIDAQABo24wbDAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAf
BgNVHSMEGDAWgBQhIhAaxTgNDjETStdzZ6cPoIpeIDArBgNVHREBAf8EITAfgh1t
cHJvLnJhbmNoZXIuazhzLmFwcHNjYWxlci5pbzAKBggqhkjOPQQDAgNHADBEAiAK
QNnQ8nhYiUuuPsybD77cKVKheZikv8UnZhsjH0/MfgIgK43EIQKhckrh0vusA8/x
yG+K6TYnF5jYQ+5wNjFVyCA=
-----END CERTIFICATE-----
which is still the dynamiclistener
let me try to delete the secret and to apply again helm upgrade
now it says i have a fake-certificate
grr
a
Check the cert manager
if it's doing anything
or if it's even setup
c
the certificate is in state not ready
Copy code
Normal  Issuing    81s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  80s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "tls-rancher-ingress-vnbzs"
  Normal  Requested  80s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-rancher-ingress
Copy code
HTTP-01" related_resource_name="cm-acme-http-solver-xwzjl" related_resource_namespace="cattle-system" related_resource_kind="Ingress" related_resource_version="v1"
E0126 21:20:39.350504       1 sync.go:190] "cert-manager/challenges: propagation check failed" err="failed to perform self check GET request '<http://mpro.rancher.k8s.appscaler.io/.well-known/acme-challenge/vp9NdadpCbmvAHOq82_WaGApqdTPoUg4MpR7pZR29AA>': Get \"<http://mpro.rancher.k8s.appscaler.io/.well-known/acme-challenge/vp9NdadpCbmvAHOq82_WaGApqdTPoUg4MpR7pZR29AA>\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" resource_name="tls-rancher-ingress-1-2631032098-2490825401" resource_namespace="cattle-system" resource_kind="Challenge" resource_version="v1" dnsName="mpro.rancher.io" type="HTTP-01"
ah i gave up my url
xd
c
lol
c
the company just got hacked, sadly and all the data was published on the dark web
that's why i want to be extra-careful
c
you’re on the right track though, the solver is trying to access the endpoint
c
do I need to open an ip address for the solver?
c
you said it was open to the world, right?
c
0.0.0.0/0 443 only
a
here is seems that it takes too long process it
oh 😄
it has to be 80
c
yes. chicken and egg problem
a
you cannot have a certificate if it cannot validate the domain
c
the solver uses http, as https won’t be available until it gets the cert.
a
but to validate it it uses http traffic not https
c
1 sec
c
it is to prove you own the hostname.
LE gives you a string, and tells you to publish it over HTTP on the host you’re trying to get a cert for. To prove that you control it.
if you can do that, and they are able to download the string from the host, they will give you a cert.
there are other challenges that do the same thing over different mediums. Updating a DNS record, for example.
c
now port 80 is open to the world
and the certificate went ready
a
Time for celebration
c
not yet
but it still says not secure
a
it says valid
c
ah nevermind it went away
THANK YOU GUYS VERY MUCH!
Indeed thank you
3 Views
Open in Slack
PreviousNext
Powered by