https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
general
  • m

    most-holiday-75301

    11/21/2022, 9:22 AM
    Setting
    CATTLE_UI_PRIMARY_COLOR
    environment variable isn't supposed to change the
    --primary
    CSS var? Although in
    /v3/settings/ui-primary-color
    it looks set, no actual color change happens (Rancher v2.6.8).
  • b

    breezy-ram-80329

    11/21/2022, 1:18 PM
    any docs on how to configure aws ELB infront of rancher server . getting too many redirects error ?
    b
    • 2
    • 2
  • b

    breezy-ram-80329

    11/21/2022, 1:29 PM
    i added use-forwarded-header = true in the ingress annotation , also configured custom header X-Forwarded-Proto = https in the nginx controller . still the ranche server is redirecting to https
    d
    • 2
    • 2
  • s

    salmon-portugal-38278

    11/21/2022, 4:11 PM
    Hello Everyone, Does anyone here has worked on the prometheus-federator project ? I have some question to ask And if there is some dedicated channels please don't hesitate to add me there ! Thank you
  • l

    lively-zoo-40381

    11/21/2022, 4:32 PM
    Hello, We have Rancher 2.5.9 with multiple RKE clusters an issue started with kubectl not responding and Rancher UI stuck In rancher Logs we see the errors below, but when we check them from the same nodes they work (connection is opened)
    rancher-7859df46-c69nz rancher 2022/11/21 16:27:56 [ERROR] error syncing 'cattle-global-data/rke-cis-1.5': handler cisBenchmarkVersionHandler: cisBenchmarkVersionHandler: Sync: error creating namespace: error while creating namespace security-scan: Internal error occurred: failed calling webhook "<http://owner.namespace.capsule.clastix.io|owner.namespace.capsule.clastix.io>": Post "<https://capsule-webhook-service.liqo.svc:443/namespace-owner-reference?timeout=30s>": dial tcp 10.43.6.146:443: connect: connection refused, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:27:58 [ERROR] error syncing 'cattle-global-data/rke-cis-1.4': handler cisBenchmarkVersionHandler: cisBenchmarkVersionHandler: Sync: error creating namespace: error while creating namespace security-scan: Internal error occurred: failed calling webhook "<http://owner.namespace.capsule.clastix.io|owner.namespace.capsule.clastix.io>": Post "<https://capsule-webhook-service.liqo.svc:443/namespace-owner-reference?timeout=30s>": dial tcp 10.43.6.146:443: connect: connection refused, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:01 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:01 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:01 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:01 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:02 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:02 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:02 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:04 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:06 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:11 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:22 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
    rancher-7859df46-c69nz rancher 2022/11/21 16:28:42 [ERROR] error syncing 'c-hjrkx/m-1526f20a73c5': handler rke-worker-upgrader: getNodePlan error for node [m-1526f20a73c5]: failed to find plan for 172.18.50.230, requeuing
  • g

    gorgeous-iron-45755

    11/21/2022, 7:09 PM
    ello everybody, We have deployed Rancher server on Kubernetes using certificates signed by our private intermediate CA, its certificate in turn being signed by our private root CA. Before deploying Rancher server, we had created the certificate secret resource for the ingress as well as the CA certificate secret resource as instructed by this guide: https://docs.ranchermanager.rancher.io/v2.5/getting-started/installation-and-upgrade/resources/update-rancher-certificate The ingress secret contains the certificate corresponding to the ingress FQDN, and the intermediate CA certificate, plus the ingress private key. The
    https://<FQDN>/v3/settings/_cacerts_
    Rancher server was deployed using the following command:
    helm install rancher rancher-stable/rancher --namespace cattle-system \
    --set hostname=<FQDN> \
    --set replicas=3 \
    --set ingress.tls.source=secret \
    --set privateCA=true
    However, when importing an existing generic Kubernetes cluster,
  • g

    gorgeous-iron-45755

    11/21/2022, 7:30 PM
    Hello everybody, We have deployed Rancher server on Kubernetes using certificates signed by our private intermediate CA, its certificate in turn being signed by our private root CA. Before deploying Rancher server, we had created the certificate secret resource for the ingress as well as the CA certificate secret resource as instructed by this guide: https://docs.ranchermanager.rancher.io/v2.5/getting-started/installation-and-upgrade/resources/update-rancher-certificate The ingress secret contains the certificate corresponding to the ingress FQDN, and the intermediate CA certificate, plus the ingress private key. Rancher server has been deployed using the following command:
    helm install rancher rancher-stable/rancher --namespace cattle-system \
    --set hostname=<FQDN> \
    --set replicas=3 \
    --set ingress.tls.source=secret \
    --set privateCA=true
    The
    https://<FQDN>/v3/settings/_cacerts_
    endpoint returns the same root CA with that contained in the corresponding secret. However, Rancher server seems not to have taken into account the ingress secret as its certificate has an invalid CN=dynamic, its signing CA certificate having CN=dynamiclistener-ca@1669050359 and being self signed. Therefore, the CA chain appears to be broken, and as a consequence, the import of an existing generic Kubernetes cluster, results in the infamous x509: certificate signed by unknown authority as shown below:
    time="2022-11-21T18:47:26Z" level=error msg="Issuer of last certificate found in chain (CN=dynamiclistener-ca@1669050359,O=dynamiclistener-org) does not match with CA certificate Issuer (CN=...,OU=...,O=...). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)"
    time="2022-11-21T18:47:26Z" level=fatal msg="Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get \"<https://rancher-demo.mas.local>\": x509: certificate signed by unknown authority"
    Can anybody help? Have we missed anything before/during the deployment?
    m
    • 2
    • 7
  • t

    thankful-energy-94761

    11/21/2022, 9:49 PM
    Hi all! So, currently, I've got a server running Docker on Ubuntu 18.04, with a bunch of docker-compose configs running, most behind Traefik. This has been a "home" lab of sorts for myself for a while. Now, I'm being asked to set up a Windows instance for Quickbooks, apparently, and so i need to create a VM, but those are harder to manage in a way that makes sense for non-technical users and i would prefer to manage containers and vms in the same place. I've heard that Rancher can do vms in docker, if so, how can I get this set up, and import my existing stuff? (Traefik is configured with Cloudflare) Noting that, we also have a second server that's yet to be set up - so I would like to connect that as well, later. -- i know that it can manage a VM host, but ideally i want to be able to have all systems be capable of both vms and containers seamlessly
    p
    w
    • 3
    • 6
  • e

    enough-balloon-64612

    11/21/2022, 10:56 PM
    Hi All, I just installed rancher on k3d/windows along with Kafka from the Apps/Charts. I'm struggling to access it from outside the cluster, can anyone drive me on the right direction? (I googled it but did not find any example)
  • b

    bumpy-printer-21267

    11/22/2022, 1:07 AM
    Is there a kubectl / rancher-cli equivalent of selecting a deployment and clicking "redeploy"? All I can find is old scripts in go/php/etc.
  • b

    bumpy-printer-21267

    11/22/2022, 1:08 AM
    And yeah, I don't want to have to change the yamls every single time, I simply want it to pull latest -latest and recreate the pods (exactly what happens when I use "Redeploy" button")
  • c

    careful-table-97595

    11/22/2022, 1:15 AM
    When will Rancher support kubernetes version 1.25+ ?
    s
    b
    • 3
    • 4
  • c

    colossal-monitor-64536

    11/22/2022, 3:10 AM
    is upgrading the local k3s not viable?
    c
    • 2
    • 5
  • p

    polite-zoo-76778

    11/22/2022, 11:09 AM
    Hi Team ! I m using Rancher v2.6.9 for creating my EKS cluster. I m having an hard time to create a managed node group In just one AZ. It seems the requirement is to have at least 2 AZ for the ASG. I change the config of the ASG in AWS to spawn the nodes in just one AZ but I have this issue now
    [Syncing error] error for cluster [c-vbbr]: health error for node group [test3] in cluster [c-vbbr]: The Amazon AutoScalingGroup eks-test3-xxxxxxxxxx has subnets ([subnet-xxxxx]) which is not expected by Amazon EKS. Expected subnets : ([subnet-xxxxxx, subnet-xxxxx, subnet-xxxx]): node group cannot be updated, must be deleted and recreated
  • p

    polite-zoo-76778

    11/22/2022, 11:10 AM
    Any idea please ? Is this possible ?
  • h

    happy-train-21560

    11/22/2022, 12:46 PM
    Hi Team! I am currently facing issues with nodes running into PLEG errors after a scheduled node restart. Does anyone know if Rancher does anything particular with the PLEG lifecycle that could be affecting this? The nodes just eventually start timing out on all their health checks which eventually causes the node to be marked as failing.
  • q

    quaint-soccer-60531

    11/22/2022, 12:46 PM
    Hi. I accidentally terminated all nodes (masters and workers) of a cluster in Rancher. All but the last master went away. But I cannot get rid of that one, it is stuck there in Removing state, since some hours. How can I recover from that state? My plan was to start a new master and populate its
    etcd
    from a snapshot of that cluster. But new nodes won't come up and stay in Registering state, probably trying to join any other master nodes that don't exist anymore.
    • 1
    • 2
  • j

    jolly-area-75887

    11/22/2022, 1:28 PM
    Cannot import AKS to rancher via rancher cli and also using rancher2 terraform
    FATA[0001] Bad response statusCode [403]. Status [403 Forbidden]. Body: [baseType=error, code=Forbidden, message=<http://clusters.management.cattle.io|clusters.management.cattle.io> "test" is forbidden: User "u-v8qr9" cannot get resource "clusters" in API group "<http://management.cattle.io|management.cattle.io>" at the cluster scope: Azure does not have opinion for this non AAD user. If you are an AAD user, please set Extra:oid parameter for impersonated user in the kubeconfig] from [<https://rancher/v3/clusters/test>]
    Can someone help with this?
  • m

    microscopic-dentist-14663

    11/22/2022, 2:21 PM
    Hey guys! I am trying to push docker image to AWS ECR.
    c
    • 2
    • 11
  • l

    limited-eye-27484

    11/22/2022, 10:13 PM
    Where should I be looking to troubleshoot if I am unable to use kubectl to view anything in a downstream managed Rancher environment? Every request I send via kubectl comes back with:
    I1122 14:06:43.519461   42615 helpers.go:264] Connection error: Get <https://my.rancher.dev/k8s/clusters/c-spjq6/api/v1/namespaces/default/pods?limit=500>: net/http: TLS handshake timeout
    Unable to connect to the server: net/http: TLS handshake timeout
    increasing to
    --v=9
    doesn’t really show any more info on why exactly this call is failing….
  • l

    limited-eye-27484

    11/22/2022, 10:14 PM
    I can browse to that rancher URL via the UI just fine, no cert errors, etc
  • l

    limited-eye-27484

    11/22/2022, 10:16 PM
    but both the local management cluster and a rancher launched RKE cluster both have the issue, but both also work just fine from the UI
  • s

    steep-furniture-72588

    11/23/2022, 12:36 AM
    Hi, Is there a way to hide the cluster management tab on rancher? I am trying to understand how to limit which users can see this option.
  • b

    boundless-intern-84896

    11/23/2022, 6:00 AM
    Hello
  • b

    boundless-intern-84896

    11/23/2022, 7:21 AM
    Hello, Is it possible to download each k8s cluster's user kubeconfig file from Rancher using API or Rancher CLI?
    a
    • 2
    • 2
  • m

    most-terabyte-51942

    11/23/2022, 8:13 AM
    Hi has anyone faced this before ? achanpur@ankit-mac ~ % /Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl start --tty=false 0 INFO[0000] Creating an instance “0” from template://default (Not from <template://0>) WARN[0000] This form is deprecated. Use
    limactl start --name=0 <template://default>
    instead FATA[0000] open /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/share/lima/examples/default.yaml: no such file or directory
    w
    • 2
    • 1
  • h

    helpful-beard-54962

    11/23/2022, 8:21 AM
    Did anyone try to use the websocket from the API to connect to the shell of a docker? It seems possible from the code and API spec, but there's no documentation on how to pass the auth tokens in the WS
    f
    • 2
    • 1
  • v

    victorious-oil-37471

    11/23/2022, 8:58 AM
    Hello, Did anyone manage to use the tls=external option when deploying a rancher through Helm in an HA architecture ? When I try to do so, the ingress correctly exposes the service from port 80 and the 3 rancher pods start up but they fail to communicate between each other with this error : "[ERROR] Failed to connect to peer wss://10.42.2.19/v3/*connect* [local ID=10.42.0.22]: websocket: bad handshake" I tried this on 2.7.0 or 2.6.9, with the compatible RKE version and had no luck with both of them. A cert-manager config with self-signed certificates works fine in the same environment but unfortunately that's not what I would like to do.
  • b

    brash-addition-38553

    11/23/2022, 9:22 AM
    Hey, we are using the new monitoring with an fresh RKE2 cluster. When we open Grafana we cant see any external nginx traffic, but only the internal traffic. Any config options we forgot? Tried using these boards.
    s
    • 2
    • 4
  • b

    breezy-ram-80329

    11/23/2022, 12:02 PM
    is there a way to run rancher cluster agents in tls verify disabled mode?
    r
    • 2
    • 14
Powered by Linen
Title
b

breezy-ram-80329

11/23/2022, 12:02 PM
is there a way to run rancher cluster agents in tls verify disabled mode?
r

red-waitress-37932

11/23/2022, 12:11 PM
sounds like something you shouldn't do without a really good reason. What do you want to accomplish with that?
b

breezy-ram-80329

11/23/2022, 12:46 PM
i have setup rancher with tls termination at load balancer level. The rancher server is redirecting to https again even though the termination happened at lb already. I guess the x-forwarded proto header has been somehow overrided by the nginx ingress controller ( according to this open issue https://github.com/kubernetes/ingress-nginx/issues/8195). due to this my cluster agents could not verify the ca cert of the rancher server (cauz i could not configure tls due to the https redirection happening) . Since this is just for poc , i need to run the agents in an insecure mode)
r

red-waitress-37932

11/23/2022, 12:47 PM
so the issue is that your LB presents the wrong cert?
so fix that 🙂
also make sure you have the "server-url" global setting set up properly
it should point to the hostname your LB serves the correct cert for
like this: https://rancher.example.com
b

breezy-ram-80329

11/23/2022, 12:50 PM
I have the set the server url to the loadbalancer dns name.
the issue is with the nginx ingress controller
via x-forwarded-proto:https header the server understands the tls termination already happend at the lb
so it won't ask for https traffic. somehow the nginx controller is not forwarding this header
r

red-waitress-37932

11/23/2022, 12:53 PM
hmmm, ok load balancers are a topic I'm only just now learning about, but from what I read, the theory is to make the LB talk directly to the services via node ports. your LB might have an ingress controller that can help with that.
what is your LB?
https://github.com/kubernetes/ingress-nginx/issues/8195#issuecomment-1324579000 someone says the issue no longer exists in ingress-nginx-4.3.0 + aws-load-balancer-controller-1.4.5. That's from the issue you posted, but it's very recent (7h ago), so I thought maybe you didn't see it yet
View count: 6