https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
w
In the absence of any reply, thought I'd try a few things - a HTTP call to port 80 on one of the k8s nodes running Rancher fails with "404 page not found", and a check on host OS with 
ss -tlnp
show nothing on port 80 listening... I then did a describe on one of the Rancher pods, and see:
Copy code
│
│ Controlled By:  ReplicaSet/rancher-759c4b7989                                                                                                                                │
│ Containers:                                                                                                                                                                  │
│   rancher:                                                                                                                                                                   │
│     Container ID:  <containerd://31eac66f8d7b59b159f10b54f6636525f755591e04d2b950ce57d8f7247b0a7>8                                                                             │
│     Image:         rancher/rancher:v2.8.0                                                                                                                                    │
│     Image ID:      <http://docker.io/rancher/rancher@sha256:cf1d155c609894c940e05bc75dd35038bb13a5aa776c3e30deb32b7d123bbf63|docker.io/rancher/rancher@sha256:cf1d155c609894c940e05bc75dd35038bb13a5aa776c3e30deb32b7d123bbf63>                                                         │
│     Port:          80/TCP                                                                                                                                                    │
│     Host Port:     0/TCP                                                                                                                                                     │
│     Args:                                                                                                                                                                    │
│       --no-cacerts                                                                                                                                                           │
│       --http-listen-port=80                                                                                                                                                  │
│       --https-listen-port=443                                                                                                                                                │
│       --add-local=true                                                                                                                                                       │
│     State:          Running                                                                                                                                                  │
│       Started:      Wed, 13 Dec 2023 22:28:48 +0000                                                                                                                          │
│     Ready:          True
So it seems like the Rancher pods are running OK, and should be listening on port 80, but do not know where the breakdown is for getting the UI up... (wish I knew more about this stuff...)
b
I'm not sure if this is your exact problem but I have run into issues with Rancher behind a nginx proxy which ended up being related to websockets. This blog post has some decent information on how to set that up, the main bit that's important is setting the headers for 
Upgrade
and 
Connection
in your site's conf:
Copy code
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen 80;
    listen [::]:80;

    server_name <HOST>;
        
    location / {
        proxy_pass <IP>;
        include proxy_params;

        # ws proxy
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}
w
Thanks for the reply @billions-vase-14972! Here is my current NGNIX config, with some internal info redacted (IP addr's, DNS names):
Copy code
# Config from: <https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/helm-chart-options#external-tls-termination>
worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

http {
    upstream rancher {
        server [redacted].104:80;
        server [redacted].105:80;
    }

    map $http_upgrade $connection_upgrade {
        default Upgrade;
        ''      close;
    }

    server {
        listen 443 ssl http2;
        server_name [redacted];
        ssl_certificate /etc/ssl/certs/rancher.pem;
        ssl_certificate_key /etc/ssl/private/rancher-lb_private.key;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Port $server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass <http://rancher>;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
            proxy_read_timeout 900s;
            proxy_buffering off;
        }
    }

    server {
        listen 80;
        server_name [redacted];
        return 301 https://$server_name$request_uri;
    }
}
As you can see, I have taken it directly off the Rancher docs site, URL in topmost comment in config.
The problem I am getting is 
ERR_TOO_MANY_REDIRECTS
in my client browser. I did a packet capture and this is what I see between the LB and the client:
Copy code
6 12.808488      [client_ip]        [nginx_lb_ip]         TLSv1.3  929    Client Hello (SNI=rancher.[redacted].com)
      8 12.809375      [nginx_lb_ip]         [client_ip]        TLSv1.3  292    Server Hello, Change Cipher Spec, Encrypted Extensions, Finished
      9 12.810329      [client_ip]        [nginx_lb_ip]         TLSv1.3  84     Change Cipher Spec, Alert (Level: Fatal, Description: Certificate Unknown)
     17 12.823759      [client_ip]        [nginx_lb_ip]         TLSv1.3  833    Client Hello (SNI=rancher.[redacted].com)
     19 12.824583      [nginx_lb_ip]         [client_ip]        TLSv1.3  292    Server Hello, Change Cipher Spec, Encrypted Extensions, Finished
     20 12.825726      [client_ip]        [nginx_lb_ip]         TLSv1.3  84     Change Cipher Spec, Alert (Level: Fatal, Description: Certificate Unknown)
     28 12.853870      [client_ip]        [nginx_lb_ip]         TLSv1.3  658    Client Hello (SNI=rancher.[redacted].com)
     30 12.861904      [nginx_lb_ip]         [client_ip]        TLSv1.3  2974   Server Hello, Change Cipher Spec, Encrypted Extensions
     31 12.861911      [nginx_lb_ip]         [client_ip]        TLSv1.3  1230   Certificate
     32 12.861930      [nginx_lb_ip]         [client_ip]        TLSv1.3  234    Certificate Verify, Finished
     34 12.863046      [client_ip]        [nginx_lb_ip]         TLSv1.3  118    Change Cipher Spec, Finished
     35 12.863165      [nginx_lb_ip]         [client_ip]        TLSv1.3  325    New Session Ticket
     36 12.863192      [client_ip]        [nginx_lb_ip]         HTTP2    146    Magic, SETTINGS[0], WINDOW_UPDATE[0]
     37 12.863206      [nginx_lb_ip]         [client_ip]        TLSv1.3  325    New Session Ticket
     38 12.863230      [nginx_lb_ip]         [client_ip]        HTTP2    125    SETTINGS[0], WINDOW_UPDATE[0], SETTINGS[0]
     39 12.863392      [client_ip]        [nginx_lb_ip]         HTTP2    536    HEADERS[1]: GET /
     41 12.864113      [client_ip]        [nginx_lb_ip]         HTTP2    85     SETTINGS[0]
     43 12.866099      [nginx_lb_ip]         [client_ip]        HTTP2    241    HEADERS[1]: 302 Found, DATA[1]
     44 12.866138      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[1] (text/html)
     46 12.869519      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[3]: GET /
     47 12.872228      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[3]: 302 Found, DATA[3]
     48 12.872269      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[3] (text/html)
     50 12.874931      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[5]: GET /
     51 12.876936      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[5]: 302 Found, DATA[5]
     52 12.876968      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[5] (text/html)
     54 12.879714      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[7]: GET /
     55 12.881672      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[7]: 302 Found, DATA[7]
     56 12.881706      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[7] (text/html)
     58 12.884488      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[9]: GET /
     59 12.886139      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[9]: 302 Found, DATA[9]
     60 12.886191      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[9] (text/html)
     62 12.889129      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[11]: GET /
     63 12.891019      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[11]: 302 Found, DATA[11]
     64 12.891059      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[11] (text/html)
     66 12.893966      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[13]: GET /
     67 12.895900      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[13]: 302 Found, DATA[13]
     68 12.895944      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[13] (text/html)
     70 12.898786      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[15]: GET /
     71 12.900778      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[15]: 302 Found, DATA[15]
     72 12.900831      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[15] (text/html)
     74 12.903734      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[17]: GET /
     75 12.905502      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[17]: 302 Found, DATA[17]
     76 12.905564      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[17] (text/html)
     78 12.908517      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[19]: GET /
     79 12.910819      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[19]: 302 Found, DATA[19]
     80 12.910900      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[19] (text/html)
     82 12.913791      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[21]: GET /
     83 12.915827      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[21]: 302 Found, DATA[21]
     84 12.915915      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[21] (text/html)
     86 12.918769      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[23]: GET /
     87 12.920919      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[23]: 302 Found, DATA[23]
     88 12.921008      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[23] (text/html)
     90 12.923812      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[25]: GET /
     91 12.925906      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[25]: 302 Found, DATA[25]
     92 12.925995      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[25] (text/html)
     94 12.929410      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[27]: GET /
     95 12.931714      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[27]: 302 Found, DATA[27]
     96 12.931810      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[27] (text/html)
     98 12.934654      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[29]: GET /
     99 12.936535      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[29]: 302 Found, DATA[29]
    100 12.936628      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[29] (text/html)
    102 12.939890      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[31]: GET /
    103 12.942059      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[31]: 302 Found, DATA[31]
    104 12.942165      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[31] (text/html)
    106 12.945487      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[33]: GET /
    107 12.947468      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[33]: 302 Found, DATA[33]
    108 12.947556      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[33] (text/html)
    110 12.950897      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[35]: GET /
    111 12.953143      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[35]: 302 Found, DATA[35]
    112 12.953234      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[35] (text/html)
    114 12.956098      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[37]: GET /
    115 12.958049      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[37]: 302 Found, DATA[37]
    116 12.958177      [nginx_lb_ip]         [client_ip]        HTTP2    85     DATA[37] (text/html)
    118 12.960983      [client_ip]        [nginx_lb_ip]         HTTP2    107    HEADERS[39]: GET /
    119 12.963162      [nginx_lb_ip]         [client_ip]        HTTP2    240    HEADERS[39]: 302 Found, DATA[39]
b
This is interesting:
Copy code
9 12.810329      [client_ip]        [nginx_lb_ip]         TLSv1.3  84     Change Cipher Spec, Alert (Level: Fatal, Description: Certificate Unknown)
I'm curious if your 
ssl_certificate_key
file with the 
.key
extension is causing issues. According to the nginx docs, the secret key should be in the PEM format, also the Rancher docs you linked referenced a pem file. Could be unrelated but worth a check at least.
w
I was wondering about that when I saw it; the LB host has its real host name as “rancher-lb” but then I CNAME’d it as “rancher” and that’s what name the cert is issued for. I did not put a SAN on the cert for “rancher-lb”.
Just checked, the .key file is in PEM format...
Copy code
-----BEGIN PRIVATE KEY-----
[redacted]
-----END PRIVATE KEY-----
Is there a way that I can test the Rancher UI access via CURL from the LB? I just don't understand why the initial GET call would be redirected so many times...
Also, if helpful, this is the cmdline that I used to install Rancher onto the 2-node K3S cluster:
Copy code
helm install \
  rancher rancher-latest/rancher \
  --version 2.8.0 \
  --namespace cattle-system \
  --set hostname=rancher.[redacted].com \
  --set bootstrapPassword=[redacted] \
  --set tls=external \
  --kubeconfig /etc/rancher/k3s/k3s.yaml

NAME: rancher
LAST DEPLOYED: Wed Dec 13 22:27:50 2023
NAMESPACE: cattle-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Rancher Server has been installed.
b
You should be able to curl the rancher ui directly with 
https://<rancher-ip>/dashboard
. There could be a few other things to check, the ingress options to use the forwarded headers depending on your nginx version, also this note in the docs:
note
If you are using a Private CA signed certificate, add 
--set privateCA=true
and see Adding TLS Secrets - Using a Private CA Signed Certificate to add the CA cert for Rancher.
w
If I used 
--set tls=external
does Rancher publish out on 443, or just 80?
b
As far as I know it would be both
I'm not an expert in this area by any means, so hopefully I don't lead you astray
w
From the LB host, I am getting 404's for both HTTP and HTTPS...
Copy code
root@rancher-lb:~# curl http://[k3s_host_running_rancher_ip]/dashboard
404 page not found
root@rancher-lb:~# curl --insecure -I https://[k3s_host_running_rancher_ip]/dashboard
HTTP/2 404
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
content-length: 19
date: Wed, 20 Dec 2023 13:55:13 GMT
@billions-vase-14972 is there someone else at Rancher that knows this better who you could pull in to help? I know it's "best effort" for us unpaid folks, but I've just followed the published Rancher docs to set up HA Rancher, and it's not working... and I'm really under the gun to get this up.
If it helps, I've also started a discussion as of a few weeks ago with Rancher sales folk for a paid support contract, but we're not there yet...
b
I will certainly ask around to get some help
Do you know if you're using traefik as the ingress controller?
b
tls=external
normally means Rancher will listen on port 80 and expects TLS to be terminated outside of the cluster somehow (on the LB)
when you perform your curl test, are you including the racnher fqdn in the host headers?
looking at this comment — https://rancher-users.slack.com/archives/C3ASABBD1/p1703080692597059?thread_ts=1702562899.481979&amp;cid=C3ASABBD1 — you are not … you want to call Rancher by URL not by IP, this is probably your issue
w
@broad-ambulance-17822 I am having a hard time understanding... If I have the following: • `k3s-node-1`\__ K8s cluster running Rancher • `k3s-node-2`/ • 
rancher-lb
(CNAME 
rancher
) - LB proxy to above Then on the NGINX LB config on rancher-lb, what do I use in the config to reach Rancher running on the K8s cluster?
My current config as shown above uses IP addr's of 
k3s-node-1
and 
k3s-node-2
b
when you install rancher, you tell it what 
server-url
to use, that should be the CNAME rancher you’re referring to
so you can circumvent the LB but ensure you have the host header info via curl … the LB is layer7, and rancher’s ingress is also layer7 they will both act upon the host you’re requesting
there’s a few ways to pass that info to curl… https://daniel.haxx.se/blog/2018/04/05/curl-another-host/
w
As seen in what I've posted above, the param I used when installing Rancher is 
--set hostname=rancher.[redacted].com
So I did set that to the hostname (CNAME) of the LB machine, which is where I'm terminating SSL/TLS
b
but the trick is, Rancher also needs that cname (if you talk directly to it, for testing, avoiding the load balancer)
w
So maybe let's roll back... Trying to install Rancher in this method: High-availability Kubernetes Install with the Helm CLI We recommend using Helm, a Kubernetes package manager, to install Rancher on multiple nodes on a dedicated Kubernetes cluster. For RKE clusters, three nodes are required to achieve a high-availability cluster. For K3s clusters, only two nodes are required. (https://ranchermanager.docs.rancher.com/pages-for-subheaders/installation-and-upgrade)
I have two K3s nodes in a cluster, and then a 3rd machine running NGINX (via Docker) to be the LB in front.
What is the right way to: • Install Rancher via Helm (what params are needed?) • What is correct NGINX LB config?
b
yea, I think I see my confusion… even if you curl 
k3s-node-1
or 
k3s-node-2
directly, going “around” the load balancer, you need to use 
<http://rancher.some-domain.com|rancher.some-domain.com>
it hits an ingress and sends to the backend rancher pods but the ingress still expects to see 
<http://rancher.some-domain.com|rancher.some-domain.com>
on port 80
w
I am just trying to verify the Rancher install on the K8s cluster is functional
✅ 1
b
your config looks good, the way you are testing it is not accurate
w
That's highly likely, as I'm really not a web-tech guy... But, what I'd expect is that if I use a browser on my network to hit the URL https://rancher.my-domain.com that I would get the Rancher UI, login page
Instead, I'm getting 
ERR_TOO_MANY_REDIRECTS
and so I'd think that the config is messed up somewhere
b
hmm I see, it could be a problem with the ingress config
when the ingress receives the request from the LB its not passing to the backend rancher pods for some reason
w
We need to find that reason then :)
b
Your packet capture shows TLS v1.3, is your ingress controller setup to use 1.3?
Copy code
6 12.808488      [client_ip]        [nginx_lb_ip]         TLSv1.3  929    Client Hello (SNI=rancher.[redacted].com)
w
I thought that SSL/TLS terminates on the LB, and then the comms between the LB and Rancher just use HTTP?
The above packet cap is between the client and the LB, not between the LB and K8s nodes
b
After some debugging I think I found a solution. I believe the issue lies with k3s and the ingress controller (this only applies if you're using the default traefik controller). You'll have to add the additional argument 
"--entryPoints.web.forwardedHeaders.insecure"
to the traefik-config by creating a `HelmChartConfig` . The file should look like:
Copy code
# ./traefik-config.yml
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
    - "--entryPoints.web.forwardedHeaders.insecure"
Then just run 
kubectl apply  -f traefik-config.yml
w
It is stock K3s, so am using Traefik... Let me give this a try.
YAAAAAAAASSSSSSS! That seems to have done it!
🎉 2
Maybe the above would be good to add to the install instructions for HA Rancher? Perhaps somewhere in this page: https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-references/helm-chart-options
Anyways thanks sooo much @billions-vase-14972, and MERRY CHRISTMAS you you and the whole Rancher gang!
b
Glad it works! I'll open an issue on the docs repo to add this. Merry Christmas!
9 Views
Open in Slack
PreviousNext
Powered by