New friday, new k3s report! https://github.com/k3s-io/k3s/discussions/6149

@fast-piano-59234 @full-painter-23916 or anyone else: I have a thread on the Amazon channel that is not getting any response. So I am posting here. I created an EKS cluster with rancher 2.6.1. I then removed a node group called pool-pvt from the EKS console. Now it looks like rancher is trying, unsuccessfully to recreate it. I am seeing API calls to cloudformation every 7 seconds. Any idea how I can fix this? I see this error in the Rancher Console:

InvalidParameterException: You cannot specify an AMI Type other than CUSTOM, when specifying an image id in your launch template. { RespMetadata: { StatusCode: 400, RequestID: "f92492c3-3f77-4f63-b91b-c6794fc81488" }, ClusterName: "pano-prod", Message_: "You cannot specify an AMI Type other than CUSTOM, when specifying an image id in your launch template.", NodegroupName: "pool-pvt" }

Hi everyone, I have a fresh install of K3S through Rancher and I'm getting the following error when trying to deploy a couple of helm charts:

failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /dev/pts/0: operation not permitted: unknown exit code: 128

Any idea what's stoping it?

hi, are there any way to run rancher from wsl1?

I have a strange problem. Try to install RKE2 on a new node using the quick installation guide. But after the installation is finished the

/var/lib/rancher/rke2/bin

folder is missing. And therefore

containerd

binary is missing and cant start. Using CentOS and installation using the

rpm

method. Someone seeing something similar?

Hey all, i'm trying to setup a rancher manager cluster. I was wondering what's the best practice between applying yaml files or doing API calls for "customizing" my installation once rancher helm chart is installed ? (add openldap auth, globalroles for example) ?

Hey everyone, as a side project I've created a Prometheus exporter for Rancher at https://github.com/David-VTUK/prometheus-rancher-exporter. It's not official, but if anyone wants to try it and provide feedback it would be very much appreciated.

How much memory should I give to a Prometheus to monitor 6 node k3s cluster using Rancher monitoring? I'm struggling to run rancher monitoring even with 6G to Prometheus as it eventually gets OOM killed. Please let me know. If there are some tips to tune the memory usage that would help. Thanks in advance!

hello, I am trying to install cluster monitoring on rancher 2.6.8 The prometheus-rancher-monitoring pod keep crashing and the events says that a secret is missing. the secret prometheus-rancher-monitoring-prometheus-tls-assets really doesn't exist but prometheus-rancher-monitoring-prometheus-tls-assets-0 does exist. why is this an issue and how can I fix this?

2m3s        Normal    Created       pod/prometheus-rancher-monitoring-prometheus-0   Created container config-reloader
2m3s        Normal    Started       pod/prometheus-rancher-monitoring-prometheus-0   Started container config-reloader
2m3s        Normal    Pulled        pod/prometheus-rancher-monitoring-prometheus-0   Container image "rancher/mirrored-library-nginx:1.21.1-alpine" already present on machine
2m3s        Normal    Created       pod/prometheus-rancher-monitoring-prometheus-0   Created container prometheus-proxy
2m2s        Normal    Started       pod/prometheus-rancher-monitoring-prometheus-0   Started container prometheus-proxy
2m2s        Normal    Killing       pod/prometheus-rancher-monitoring-prometheus-0   Stopping container prometheus
2m2s        Normal    Killing       pod/prometheus-rancher-monitoring-prometheus-0   Stopping container config-reloader
2m2s        Normal    Killing       pod/prometheus-rancher-monitoring-prometheus-0   Stopping container prometheus-proxy
2m          Normal    Scheduled     pod/prometheus-rancher-monitoring-prometheus-0   Successfully assigned cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0 to ip-10-102-66-114.ec2.internal
2m          Normal    Pulled        pod/prometheus-rancher-monitoring-prometheus-0   Container image "<http://quay.io/prometheus-operator/prometheus-config-reloader:v0.56.0|quay.io/prometheus-operator/prometheus-config-reloader:v0.56.0>" already present on machine
2m          Normal    Created       pod/prometheus-rancher-monitoring-prometheus-0   Created container init-config-reloader
2m          Normal    Started       pod/prometheus-rancher-monitoring-prometheus-0   Started container init-config-reloader
117s        Normal    Scheduled     pod/prometheus-rancher-monitoring-prometheus-0   Successfully assigned cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0 to ip-10-102-66-114.ec2.internal
117s        Normal    Pulled        pod/prometheus-rancher-monitoring-prometheus-0   Container image "<http://quay.io/prometheus-operator/prometheus-config-reloader:v0.56.0|quay.io/prometheus-operator/prometheus-config-reloader:v0.56.0>" already present on machine
117s        Normal    Created       pod/prometheus-rancher-monitoring-prometheus-0   Created container init-config-reloader
116s        Normal    Started       pod/prometheus-rancher-monitoring-prometheus-0   Started container init-config-reloader
114s        Normal    Scheduled     pod/prometheus-rancher-monitoring-prometheus-0   Successfully assigned cattle-monitoring-system/prometheus-rancher-monitoring-prometheus-0 to ip-10-102-66-114.ec2.internal
50s         Warning   FailedMount   pod/prometheus-rancher-monitoring-prometheus-0   MountVolume.SetUp failed for volume "tls-assets" : secret "prometheus-rancher-monitoring-prometheus-tls-assets" not found

It's the permissions mechanism of rancher (rbac base)

Love the new look and feel of the site 👍🏻

Hello everyone 👋 , I hope I’m in the right place, but I have a question about network partition and would really appreciate if someone could help: https://github.com/k3s-io/k3s/discussions/6154

Hello, I have a problem with rancher after the upgrade to 2.6.6 : impossible to make a

kubectl

using the kubeconfig in another computer for two specific clusters (olders EKS cluster saws as ). Both clusters are in 1.21 version. I have this error after all `kubectl`/`helm` commands :

Error from server (InternalError): an error on the server ("unable to create impersonator account: ClusterUnavailable 503: ClusterUnavailable 503: cluster not found") has prevented the request from succeeding

One other weird thing : I can't use execute shell or logs directly in rancher GUI for this two clusters. It works fine before the 2.6.6. I decide to upgrade to rncher 2.6.8. But, only for this two clusters, the cluster-agent upgrade is not propaged (I'm still in clter-agent 2.6.6 for them) After navigate in rancher console, I saw, one of this clusters in error in cluster management page. It said the cluster is unavailable (But I can navigate and manipulate it in rancher interface). I research for it and I found this command :

kubectl patch <http://clusters.management.cattle.io|clusters.management.cattle.io> <REPLACE_WITH_CLUSTERID> -p '{"status":{"agentImage":"dummy"}}' --type merge

I try this for the "unavailable" cluster but no changes. I decide to watch the logs in the rancher deployment and I get this error :

[ERROR] [secretmigrator] failed to migrate service account token secret for cluster c-wrh8l, will retry: Operation cannot be fulfilled on <http://clusters.management.cattle.io|clusters.management.cattle.io> "c-wrh8l": the object has been modified; please apply your changes to the latest version and try again

Somebody can help me ?

Hello Everybody, Pressed Enter to soon. Please ignore.

Hello Everybody, I am trying to run Rancher Desktop on Windows 10. This is a corporate desktop and I am behind a firewall. Rancher Desktop starts up and within a few seconds I get the error message: Could not fetch releases: Proxy Authorization Required

(node:14232) UnhandledPromiseRejectionWarning: FetchError: invalid json response body at <https://desktop.version.rancher.io/v1/checkupgrade> reason: Unexpected token < in JSON at position 0
    at C:\Users\sau\AppData\Local\Programs\Rancher Desktop\resources\app.asar\node_modules\node-fetch\lib\index.js:273:32
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Xn.checkForUpdates (C:\Users\sau\AppData\Local\Programs\Rancher Desktop\resources\app.asar\dist\app\background.js:29:58509)
    at async Xn.getLatestVersion (C:\Users\sau\AppData\Local\Programs\Rancher Desktop\resources\app.asar\dist\app\background.js:29:60057)
    at async NsisUpdater.getUpdateInfoAndProvider (C:\Users\sau\AppData\Local\Programs\Rancher Desktop\resources\app.asar\node_modules\electron-updater\out\AppUpdater.js:298:19)
    at async NsisUpdater.doCheckForUpdates (C:\Users\sau\AppData\Local\Programs\Rancher Desktop\resources\app.asar\node_modules\electron-updater\out\AppUpdater.js:312:24)
    at async ci (C:\Users\sau\AppData\Local\Programs\Rancher Desktop\resources\app.asar\dist\app\background.js:29:63495)
(Use `Rancher Desktop --trace-warnings ...` to show where the warning was created)
(node:14232) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see <https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode>). (rejection id: 2)
Config file has no clusters, will retry later
[21072:0919/114238.364:ERROR:<http://gpu_init.cc|gpu_init.cc>(446)] Passthrough is not supported, GL is disabled, ANGLE is

As per the message it appears that the Rancher Desktop software is being blocked by the proxy while attempting to download some data. What is the destination IP/Hostname is Rancher Desktop trying to access so that I may ask my IT team to relax the restrictions for that host. Thanks, Sau

Hi everyone, I have installer Rancher 2.6.8 recently. I setup my SSL with lets encrypt but no ingress. I created manually a certificate as follows:

apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Certificate
metadata:
  name: <http://rancher.sand.example.com|rancher.sand.example.com>
  namespace: istio-system
spec:
  privateKey:
    rotationPolicy: Always
  secretName: <http://rancher.sand.example.com|rancher.sand.example.com>
  commonName: <http://rancher.sand.example.com|rancher.sand.example.com>
  issuerRef:
    name: letsencrypt-prod-istio
    kind: ClusterIssuer
  dnsNames:
  - <http://rancher.sand.example.com|rancher.sand.example.com>

The cluster already have istio installed so I created the following Virtual Service and Gateway:

apiVersion: <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
kind: Gateway
metadata:
  name: rancher
  namespace: cattle-system
spec:
  selector:
    app: istio-ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
      - <http://rancher.sand.example.com|rancher.sand.example.com>
      tls:
        mode: SIMPLE
        credentialName: <http://rancher.sand.example.com|rancher.sand.example.com>

---
apiVersion: <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
kind: VirtualService
metadata:
  name: rancher
  namespace: cattle-system
spec:
  gateways:
  - rancher
  hosts:
  - <http://rancher.sand.example.com|rancher.sand.example.com>
  http:
  - name: "http"
    route:
    - destination:
        host: rancher.cattle-system.svc.cluster.local
        port:
          number: 80

Everything works except when I try under my terminal

kubectl exec

and

kubectl port-forward

.

$ kubectl exec -v=7 -it myPod -- bash
I0919 16:30:52.730382   61542 round_trippers.go:457] Response Status: 403 Forbidden in 78 milliseconds
I0919 16:30:52.730998   61542 helpers.go:216] server response object: [{
  "metadata": {}
}]
F0919 16:30:52.731059   61542 helpers.go:115] Error from server:

Has anyone has this issue before?

Hi All, Could anyone help me debug MetalLB and RKE2? I believe I have metallb installed and configured to hand out IPs in the 192.168.1-240-245 range, but I'm not able to hit the external ip from outside the cluster so I think I'm missing a configuration somewhere. Setup: • I'm using Ubuntu 20.04 as my host and I've deployed 3 RockyLinux8 VMs using Vagrant. One VM is my server, the other two are agents. I'm using Cilium as the CNI. After installing RKE2, kubectl get nodes and pods look good- no obvious failures or errors • I deployed MetalLB using the manifest and configured its IP pool to give out in the range of 192.168.1.240-245 with L2Adertisement • I deployed a vanilla nginx container to my cluster and a loadbalancer service for it and I see that the service was assigned an external ip of 192.168.1.240 Test: • I launched a busybox container on to the cluster. From it I'm able to wget the default homepage of my nginx container, using both the Cluster-IP and External-IP. So I believer MetalLB is working to some level. Problem: • I'm not able to wget the default nginx homepage from my Host Machine using the External-IP. What am I missing? This should be possible right?

Hello everyone - wondered if anyone could help me. On a Windows 11 box. Running nerdctl build on anything gives me: /usr/local/bin/docker-credential-rancher-desktop: source: line 5: can't open '/etc/rancher/desktop/credfwd': No such file or directory I changed congif.json with what I could find online - changed credsStore to credStore. took out wincred, etc. I have uninstalled and reinstalled rancher desktop multiple time, reset to factory, etc. Any ideas on what to try?

^^ asking in rancher desktop

found the answer - Thank you!

Hi, I couldn't find any site describing how to deploy rancher on kubernetes with cri-o. I get error: Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:} I couldn't find any solution to deploy it with configuration: kubernetes(1.24)+cri-o ; no matter which certificate I chose eac time I get this message and Rancher crushes

after creating a service account, I am not able to access the RKE cluster, I get the following error- You must be logged in to the server (the server has asked for the client to provide credentials (get pods)), I created the service account, created the role bindings, get the secret, got the TOKEN, did set the context with the new SA, still this error? why?

The cluster was created by rancher running on k3s

👋 likely due to being new to containers in general, but I can't seem to get rancher desktop to use a local mount when I specify it. The location exists in the already shared /Users/$USER directory, but the containers don't seem to actually write to that directory, thus resetting back to defaults when I restart them or update them, it seems like the default lima config prevents this from being writable, but I can't find any other mounted location. I tried removing rancherdesktop, then building the same setup in lima on it's own and that worked, but ran into some port issues that rancherdesktop solved for me (still not sure on that one tbh). Any advice would be appreciated. macOS m1 as a host.

Hi All, Can we access Rancher Server UI, if we deploy it in Ubuntu VM with docker install method. Private DNS pointed to Private IP of the Ubuntu VM. I wants to access Rancher Ui with private DNS rather than public Ip of server

@brash-planet-10109 I think it should be possible - currently for tests I have VM with docker and Rancher in internal network and I can connect without any problems ( I didn't set up DNS , but I assume if it works on private network than it should work on private hostname too)

@creamy-crowd-89310 are you able to access Rancher UI with private ip of that VM?

yes

@creamy-crowd-89310 Is there any specific configuration you did for this to work. If not, I will just ask our sec team. That I need my VM private IP must be accessible with 8080 port. Like http://privateIP:8080