Hi, we want to host rancher in an internal network, with no connections from the internet possible. In addition to internal vSphere providers, we also want to use external providers like Linode, Azure or Digital Ocean. Now I see a problem: According to the Port Requirements page in the manual, the Kubernetes clusters would need to be able to connect back to the Rancher in my internal network. This would weaken our network security majorly. To comply with out guidelines, I would have to move Rancher to a DMZ. This would mean it can no longer make connections to the Kubernetes API endpoints on our internal clusters on vSphere, as well as the vSphere API itself. Is there a way to run Rancher without requiring incoming connections like that?

Hello, I am running Rancher Desktop 1.7 and I am trying to enable experimental features to run

rdctl set --experimental.virtual-machine.type vz

. I tried following the instructions here to enable it from the UI, but it isn't working as I don't have a global settings button on the top left. Can anyone help?

Hello all , Can some one share how to setup pods antiAffinity from Rancher UI ..

👋 Hi everyone, is this the channel for asking errors related questions?

Hi Everyone! I have some questions regarding the rancher monitoring helm chart. I've set this up with the thanos sidecar running and grafana as an interface. Is it possible to set up the thanos querier/store within the same helm chart? If not, what would be the best practice of setting up the thanos querier to work with the rancher monitoring setup.

I have a downstream k3s cluster that had issues with its rancher-webhook chart as the upstream rancher instance was upgraded beyond 2.7 - I’ve now gotten that chart to the point where helm has a “deployed” release, and annotated the webhook service so that it received a necessary tls secret, but the helm chart is an older version than that in other downstream clusters I run - what’s the best way to force a deploy of the correct version of rancher-webhook here?

Hello. how do the rancher agent determine what to put into the server: url on the file /etc/rancher/rke2/config.yaml.d/50-rancher.yaml every time the rancher agent is restarted, it writes this file. and even after i have removed and recreated all nodes and removed and recreated the cluster in rancher gui, the server: address becomes a ipv6 address that is not on this network any more.

can there be some leftover remnants of an old cluster on the rancher server ?

Good afternoon! I'm in some dire need of guidance on having our Rancher's host name certificate ( for tls-rancher-ingress ) re-generated, it has expired and I'm unable to get it renewed. I've identified 2 possible reasons for it, being. 1 I suspect this is because of it not being able to communicate internally due to seeing the following:

kubectl logs --timestamps  -n cert-manager cert-manager-webhook-5d4fd5cb7f-mq94z | tail -5
2023-05-24T14:20:58.529881904+02:00 I0524 12:20:58.513458       1 logs.go:59] http: TLS handshake error from 89.233.X.X:34022: read tcp 10.42.1.196:10250->89.233.X.X:34022: read: connection reset by peer
2023-05-24T14:20:58.529886873+02:00 I0524 12:20:58.514768       1 logs.go:59] http: TLS handshake error from 89.233.X.X:34028: EOF
2023-05-24T14:20:58.538134400+02:00 I0524 12:20:58.522918       1 logs.go:59] http: TLS handshake error from 89.233.X.X:34058: read tcp 10.42.1.196:10250->89.233.X.X:34058: read: connection reset by peer
2023-05-24T14:20:58.557263869+02:00 I0524 12:20:58.524215       1 logs.go:59] http: TLS handshake error from 89.233.X.X:34034: EOF
2023-05-24T14:20:58.563356287+02:00 I0524 12:20:58.558927       1 logs.go:59] http: TLS handshake error from 89.233.X.X:34042: read tcp 10.42.1.196:10250->89.233.X.X:34042: read: connection reset by peer

Is there any way to get the

--insecure-skip-tls-verify

functionality into the command(s) that need executing? --- 2 It's trying to use 'rancher' as it's Issuer, where as it should/could be using the ClusterIssuer named "*letsencrypt-production*".

kubectl cert-manager -n cattle-system status certificate tls-rancher-ingress
Name: tls-rancher-ingress
Namespace: cattle-system
Created at: 2023-05-24T14:20:57+02:00
Conditions:
  Issuing: True, Reason: DoesNotExist, Message: Issuing certificate as Secret does not exist
  Ready: False, Reason: DoesNotExist, Message: Issuing certificate as Secret does not exist
DNS Names:
- cluster.domain.ext
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    19m   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  19m   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "tls-rancher-ingress-xxwck"
  Normal  Requested  19m   cert-manager-certificates-request-manager  Created new CertificateRequest resource "tls-rancher-ingress-8wb4j"
error when getting Issuer: <http://issuers.cert-manager.io|issuers.cert-manager.io> "rancher" not found
error when finding Secret "tls-rancher-ingress": secrets "tls-rancher-ingress" not found
Not Before: <none>
Not After: <none>
Renewal Time: <none>
CertificateRequest:
  Name: tls-rancher-ingress-8wb4j
  Namespace: cattle-system
  Conditions:
    Approved: True, Reason: <http://cert-manager.io|cert-manager.io>, Message: Certificate request has been approved by <http://cert-manager.io|cert-manager.io>
  Ready: False, Reason: Pending, Message: Referenced "Issuer" not found: <http://issuer.cert-manager.io|issuer.cert-manager.io> "rancher" not found
  Events:
    Type    Reason              Age   From                                                Message
    ----    ------              ----  ----                                                -------
    Normal  WaitingForApproval  19m   cert-manager-certificaterequests-issuer-ca          Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  19m   cert-manager-certificaterequests-issuer-selfsigned  Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  19m   cert-manager-certificaterequests-issuer-venafi      Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  19m   cert-manager-certificaterequests-issuer-vault       Not signing CertificateRequest until it is Approved
    Normal  WaitingForApproval  19m   cert-manager-certificaterequests-issuer-acme        Not signing CertificateRequest until it is Approved
    Normal  <http://cert-manager.io|cert-manager.io>     19m   cert-manager-certificaterequests-approver           Certificate request has been approved by <http://cert-manager.io|cert-manager.io>
    Normal  IssuerNotFound      19m   cert-manager-certificaterequests-issuer-selfsigned  Referenced "Issuer" not found: <http://issuer.cert-manager.io|issuer.cert-manager.io> "rancher" not found
    Normal  IssuerNotFound      19m   cert-manager-certificaterequests-issuer-ca          Referenced "Issuer" not found: <http://issuer.cert-manager.io|issuer.cert-manager.io> "rancher" not found
    Normal  IssuerNotFound      19m   cert-manager-certificaterequests-issuer-vault       Referenced "Issuer" not found: <http://issuer.cert-manager.io|issuer.cert-manager.io> "rancher" not found
    Normal  IssuerNotFound      19m   cert-manager-certificaterequests-issuer-acme        Referenced "Issuer" not found: <http://issuer.cert-manager.io|issuer.cert-manager.io> "rancher" not found
    Normal  IssuerNotFound      19m   cert-manager-certificaterequests-issuer-venafi      Referenced "Issuer" not found: <http://issuer.cert-manager.io|issuer.cert-manager.io> "rancher" not found

--- Anyone that could guide me in the right direction? I'm bit blindly trying search terms and there results but unsuccessful so far.

Hello, I don't know how to create a production stack with a pod with headless wordpress with a front end coded in react and headless wordpress + mariab and another pod with an exposed mongodb instance. I also want to know how to use certificate from lets encrypt in k3s. please help me, I am a beginner in this field 😉

Hi all. Having trouble with a fleet deployment of cert manager. I keep getting an error:

Chart requires kubeVersion: >= 1.21.0-0 which is incompatible with Kubernetes v1.20.0

However I my cluster is using v1.25.5+k3s1. I saw an issue on fleet that said to fix this upgrade to a more recent version of fleet. I tried updating with helm to 0.7.0-rc3 but I am seeing

rancher/fleet:v0.6.0

and

rancher/fleet-agent:v0.6.0

when I look at my pods.

➜  ~ helm -n cattle-fleet-system install --create-namespace --wait \
    fleet <https://github.com/rancher/fleet/releases/download/v0.7.0-rc.3/fleet-0.7.0-rc.3.tgz>
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/patrick/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/patrick/.kube/config
NAME: fleet
LAST DEPLOYED: Tue May 23 23:05:55 2023
NAMESPACE: cattle-fleet-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Not sure why this isn't updated.

I'm having problems getting a K3s cluster (1.23.6+k3s1) connected to a Rancher (2.7.1) instance. It was previously attached to an older Rancher instance, was Rancher instance using the Rancher UI, then added to the new cluster (via the UI). When adding it to the new cluster, everything seems to go smoothly except the ClusterAgent never fully connects. When I look at the logs for the cluster-agent pod, this is what I see during the initial few starts:

INFO: <https://REDACTED/ping> is accessible
INFO: REDACTED resolves to REDACTED
time="2023-05-24T13:45:38Z" level=info msg="Listening on /tmp/log.sock"
time="2023-05-24T13:45:38Z" level=info msg="Rancher agent version v2.7.1 is starting"
time="2023-05-24T13:45:38Z" level=fatal msg="looking up cattle-system/cattle ca/token: failed to find service account cattle-system/ca
ttle: serviceaccounts \"cattle\" is forbidden: User \"system:serviceaccount:cattle-system:cattle\" cannot get resource \"serviceaccoun
ts\" in API group \"\" in the namespace \"cattle-system\""

After a few pod crash/restarts, it calms down and I see this:

time="2023-05-24T13:46:50Z" level=info msg="Listening on /tmp/log.sock"
time="2023-05-24T13:46:50Z" level=info msg="Rancher agent version v2.7.1 is starting"
time="2023-05-24T13:46:50Z" level=info msg="Connecting to <wss://REDACTED/v3/connect/register> with token starting with
[REDACTED]"
time="2023-05-24T13:46:50Z" level=info msg="Connecting to proxy" url="<wss://REDACTED/v3/connect/register>"

When I enable debug logging in the pod, I see an occasional "Wrote ping" message, but not much else. Any ideas on where to start looking?

👀

Where do I find the Ingress Controller used in my Rancher setup? We seem to be using Traefik and default is NGINX from this link but i can’t find any Traefik deployments in the Rancher dashboard

We have Rancher hosted on a k8s cluster. No airgapped challenges. On April 15 this year, the rancher chart was given upgrade instruction, it appears to have failed and remained in a Pending state. A roll back was attempted and now the Cattle-System : rancher release is stuck in "Pending-Rollback". Has anyone seen similar issues, overcome them, would like to share how the issue was corrected?

Hi all, I have a Rancher v2.7.3 installed and many K3s clusters imported, some v1.21.5+k3s2 and some v1.25.6+k3s1. It is not clear to me how to enable from RancherUI the automatic upgrade of k3s, since in the docs (Automated Upgrades | K3s) it is specified to not install the system-upgrade-controller on each K3s cluster if they are imported into Rancher. Can you please help me to understand how to proceed? Thanks

Hello all ! I have a Rancher cluster and multiple downstream clusters, and I need to generate SSL certificates for each cluster using the Let's Encrypt API with the following DNS wildcard formats: •

*.<http://local.rke.example.com|local.rke.example.com>

•

*.<http://downstream1.rke.example.com|downstream1.rke.example.com>

•

*.<http://downstream2.rke.example.com|downstream2.rke.example.com>

•

*.<http://downstream3.rke.example.com|downstream3.rke.example.com>

However, the Let's Encrypt production API has a limit of 50 certificates per registered domain per week. Since the registered domain is

<http://example.com|example.com>

, I may quickly reach this limit. I'm not sure about how the renewal process for certificates works if I decide to use the ACME Terraform provider. Specifically, I'm uncertain if renewing the certificates requires executing an additional Terraform apply. I want to know if there is another solution to achieve this please ?

unless things have changed, let's encrypt used to specifically reject example.com. If you have a private CA for example.com, why not generate your own certificates and use those?

👋 Hi everyone! I've done a search but can't find a specific answer to this -... I'm doing a POC around switching our teams from docker desktop to potentially Rancher. There will be a period of time where we're using both across different machines - so we can't be changing code to work with one, that then breaks the other - ad-infinitum!! on our dev machines, in DD, we reference local files on the host via

/run/desktop/mnt/host/c/some_path

there seems to be some sort of mapping in RD too - but is there a way I can have a generic config that maps the above for all pods? so we don't need to update the reference in every location when switching between ? ta

Any one is getting this error after upgrading to 2.7.3

Unknown error: <http://samltokens.management.cattle.io|samltokens.management.cattle.io> "xxxxx" not found

Hi all, trying to provision nodes with Rancher v2.5.9 and v2.7.1 and its built-in node-driver for Digital Ocean we observe a very unstable behavior with following errors:

Error creating machine: Error running "sudo apt-get update": ssh command error: command: sudo apt-get update err: exit status 100 output: Hit:1 <http://archive.ubuntu.com/ubuntu> jammy InRelease

[ERROR] handler node-controller: Error creating machine: Error installing Docker: , requeuing

[ERROR] handler node-controller: Error creating machine: Error running "sudo apt-get update": ssh command error: command: sudo apt-get update err: exit status 100 output: Get:1 <http://security.ubuntu.com/ubuntu> focal-security InRelease [114 kB], requeuing

Does anyone have an idea if it could be the node-driver or on Digital Ocean's side? Or are there other ways to debug the problem besides looking through the Rancher-logs?

Asking again, has anyone dealt with this successfully, i.e. barring a rip and reinstall? https://rancher-users.slack.com/archives/C3ASABBD1/p1684948186808469

Anyone using Istio service mesh instead of default Traefik? I am interested in knowing the benefits of Istio over Traefik

Is there any documentation about what the default logging structure is for the rancher logging operator? I just want to get a general schema of the data being passed

Hi - We have a rancher setup running as docker with few eks clusters managed by it. We are migrating the host to another aws account and the domain is getting changed as well. Now when the new host came up all the clusters show up as unaivailable saying cluster agent is not connected . Can anyone please suggest on how we can get this fixed

GOOD MORNING EVERYONE, BUENOS DIAS, I am trying to set up a windows node following these steps . The video didn't go into much detail setting up the node only registering it to the cluster. My node is running a fresh install of Windows Server 2019 and I'm not sure if it's missing anything, but when I run the registration command generated from the Registration page, my cluster sees the node but gets stuck on "Waiting for probes: kubelet" . Am I missing something on my windows server installation?

What exactly does the State field in the rancher GUI pick up on? I have some custom resources that are ostensibly fine, but Rancher displays them as “Error” state

Hi all. Does anyone have any tips on ways of automating impersonation in rancher? Something like an operator that would temporarily grant a rolebinding to a user for break glass type privilege escalation.

Hi. I was running Rancher v2.7.3 on a Single Node with Docker. The web interface went down suddenly. Restarting the container shows errors like these below. I wasn't able to find the reason for the failure. Any hint?

time="2023-05-27T15:25:50Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
E0527 15:25:55.611263     101 controller.go:163] Error removing old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
time="2023-05-27T15:25:55Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
time="2023-05-27T15:25:57Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
        error creating chain "KUBE-IPTABLES-HINT": exit status 3: Ignoring deprecated --wait-interval option.
E0527 15:25:59.599063     101 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"a1861d6e3f719d7af4534bce849a6b710b8aeec31ebc26efa1b499eb57ed0a64\": not found" podSandboxID="a1861d6e3f719d7af4534bce849a6b710b8aeec31ebc26efa1b499eb57ed0a64"
E0527 15:26:00.166615     101 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"ccaa4af90b9d268420d713770e59e0cddd6a5df90cf6351ac73e18e552d56fd5\": not found" podSandboxID="ccaa4af90b9d268420d713770e59e0cddd6a5df90cf6351ac73e18e552d56fd5"
Trace[108069360]: ---"Write to database call finished" len:2979,err:Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=10s>": context deadline exceeded 10003ms (15:26:09.987)
time="2023-05-27T15:26:09Z" level=warning msg="Failed to create Kubernetes secret: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"<https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=10s>\": context deadline exceeded"
Trace[1354403927]: ---"Write to database call finished" len:326,err:Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=10s>": context deadline exceeded 10000ms (15:26:10.169)
time="2023-05-27T15:26:10Z" level=warning msg="Error ensuring node password secret for pre-validated node 'local-node': Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"<https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=10s>\": context deadline exceeded"
        error checking rule: exit status 2: Ignoring deprecated --wait-interval option.
E0527 15:26:10.367605     101 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"ce831e413c98914abb6b3b94bfec7beae966a72ca73be22501da503d6a8eb095\": not found" podSandboxID="ce831e413c98914abb6b3b94bfec7beae966a72ca73be22501da503d6a8eb095"
W0527 15:26:13.073302     101 dispatcher.go:174] Failed calling webhook, failing open <http://rancher.cattle.io.features.management.cattle.io|rancher.cattle.io.features.management.cattle.io>: failed calling webhook "<http://rancher.cattle.io.features.management.cattle.io|rancher.cattle.io.features.management.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s>": proxy error from 127.0.0.1:6443 while dialing 10.42.0.52:9443, code 503: 503 Service Unavailable
E0527 15:26:13.073346     101 dispatcher.go:181] failed calling webhook "<http://rancher.cattle.io.features.management.cattle.io|rancher.cattle.io.features.management.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s>": proxy error from 127.0.0.1:6443 while dialing 10.42.0.52:9443, code 503: 503 Service Unavailable
E0527 15:26:31.769067     101 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"fleet-controller\" with CrashLoopBackOff: \"back-off 10s restarting failed container=fleet-controller pod=fleet-controller-55c547c6d5-47jjd_cattle-fleet-system(949dc5b1-1aff-4c5f-912d-8f1b37793c68)\"" pod="cattle-fleet-system/fleet-controller-55c547c6d5-47jjd" podUID=949dc5b1-1aff-4c5f-912d-8f1b37793c68

Does anyone have knowledge on configuring nginx-ingress on a locally running RKE2 HA cluster with an external IP, so that this IP can be used by others on the same network to access the cluster services? I would appreciate any guidance or assistance.