Hi all, anyone mess with air gap installs? Is there any faster way to populate a private repository instead of using the two provided scripts save and load images? And is it supposed to have so many images 340-ish?

How are you dealing with RKE1 cluster deploy and the docker install need when there is no access to the Internet? Was able to create private registry and install all via air gap means but that part was not covered in the instructions. Any way to create something local? How are folks dealing with this in environments with no internet access?

Hi folks, something went bad on creating a new cluster (2 hosts, RKE): Rancher claims "Provisioning" on the Cluster Management page for the new one, but there is a red bar saying "namespaces "c-mflh9" not found". Trying to delete the cluster I get an error popup saying

{
  "type": "error",
  "links": {},
  "code": "NotFound",
  "message": "<http://clusters.provisioning.cattle.io|clusters.provisioning.cattle.io> \"c-mflh9\" not found",
  "status": 404
}

Hopefully a very quick question about everyone's favorite topic: SSL! [Windows] Rancher Desktop starts up but can't reach the internet as my company uses TLS inspection. Is it possible to configure Rancher or the underlying technology to trust a custom root CA?

Hi, does anybody know how to set up apps for air gapped env? I need to deploy it in artifactory, but I couldn't find how to do that.

Hi folks, I upgraded rancher from 2.6.9 to 2.7.0 and somehow longhorn disappeared from the available charts on my local cluster. Other official charts like logging and monitoring are still here. Any ideas ?

Hey friends, I'm trying to play with k3s with qemu/alpine and I'm hitting a failure at startup and I'm trying to figure out what exactly is causing the issue. When I try to init the cluster I get an error like this:

# k3s server --cluster-init
FATA[0000] starting kubernetes: preparing server: start managed database: listen tcp 127.0.0.1:2380: bind: cannot assign requested address

Seems like most folks hit this issue trying to run in docker, which the resolution is to open the external port. I've tried adding similar settings in qemu to forward the port to the host, but that has not improved anything. It seems lightly weird to me that I'd need to expose ports to just run a single instance, but I'm not well versed in any of this tech, so eh. I'm not sure what next to try or where to look for other logs for more information. I've tried increasing the log output, but that did not add any information thus far.

Hi having trouble with retaining data in the pod. I have installed rancher desktop and created a Kubernetes pod. I copied/saved data in the pod (created a folder "*app"* and saved data in it). When I exit rancher desktop and switch off my PC and restart rancher desktop next day all the data is lost which I saved inside the pod. Can any one please help.

how can we reimport existing EKS cluster as generic one?. We want to reimport our EKS cluster as generic one. Our doubt - rancher might end up deleting EKS cluster? we just want to detach it from rancher and reimport same as

generic

one.

looks like

--k3s-server-arg

no longer provided for

k3d

? what is the replacement for it?

அ  ~  k3d cluster create --k3s-server-arg "--no-deploy=traefik" --agents 2 -v "/home/bsivasub/kube:/kube@agent[0,1]" -v "/home/bsivasub/kube:/kube@server[0]" --port "80:80@loadbalancer"

FATA[0000] unknown flag: --k3s-server-arg

Hi Friends, Hi Team, One of the glusterfs filesystem has reached 100%. How can I fix this? /dev/mapper/vg_cbbf80a61dea3e7812cacb44b9ff397e-brick_8bb1367b1f384fb9f7fc719455b73959 5.0G 5.0G 24K 100% /var/lib/heketi/mounts/vg_cbbf80a61dea3e7812cacb44b9ff397e/brick_8bb1367b1f384fb9f7fc719455b73959 [root@sn1ylpwku0001 ~]# cat /etc/fstab | grep brick_8bb1367b1f384fb /dev/mapper/vg_cbbf80a61dea3e7812cacb44b9ff397e-brick_8bb1367b1f384fb9f7fc719455b73959 /var/lib/heketi/mounts/vg_cbbf80a61dea3e7812cacb44b9ff397e/brick_8bb1367b1f384fb9f7fc719455b73959 xfs rw,inode64,noatime,nouuid 1 2

rke2 cluster struck at provisioning on harvester cluster. #7813

Hi All, i installed OPA gatekeeper on a downstream cluster from Rancher UI. The requirement is to create a custom OPA policy template which matches two arrays. The arrays are definition of port numbers. For simplicity sake, i have the following:

import <http://future.keywords.in|future.keywords.in>

violation[{"msg": msg}] {
    allowedPorts := [8161, 9404, 61616, 5445, 9999, 2022]
    actualPorts := [8161, 9404, 61616, 5445]

	some port in actualPorts; not port in allowedPorts
    msg := sprintf("service has forbidden exposed ports: %v", [actualPorts])
}

the idea is that if a port in

actualPorts

doesn't match any in

allowedPorts

, it should result in a violation. this works fine on the OPA playground. but it doesn't work when used as OPA template in Rancher. it seems the

import

doesn't work. i've tried

import data.lib.future.keywords

and

import '<http://github.com/open-policy-agent/opa/internal/future|github.com/open-policy-agent/opa/internal/future>'

, Q> any ideas on how i can compare the arrays or import the required module?

Hi, I am trying to setup kube prometheus stack on K3s and I am having a lot of issues. By default kube controller, proxy and etcd services are not mapped to any container (with the latest version of the stack) and I have empty dashboards for those services in Grafana. Also I read that K3s is exposing metrics from all these in each of them so for the ones which work I will get duplicated data in the Prometheus. So is there custom version of kube prometheus stack which works with K3s and if not, are there any workarounds to get it to work with the latest prom stack version?

Hey folks, I'm trying to find the "right" path/way to setup grafana, prometheus, telegraf and monitoring. I have a bunch of services which can either talk to telegraf and push metrics or they offer the monitoring endpoint for prometheus themselves (doesn't really matter ultimately) but I'm a bit confused of how to do it right: I'm using Rancher, Rancher has a Monitoring section with a lot of Kubernetes monitoring and an existing Prometheus and Grafana but My main questions and looking for help are: • I'm afraid of using that also for my app specific dashboards and metrics collection, is this wrong and i should use the Rancher/Cattle-Monitoring stuff? • Do I install Telegraf, Prometheus, Grafana on my own and crawl the kubernetes prometheus from that instance to build a unified dashboard? • If I don't use Telegraf but the typical /metrics endpoint, how do i configure prometheus / k8s manifests to automatically collect the data? I found really a lot of stuff in regards to Grafana, Prometheus, Rancher Monitoring and such but not really a good/best practices guide for monitoring Kubernetes & Applications/Services on top of Kubernetes. I would appreciated any hints 🙂. Thanks! 🙏

Greetings. I'm new to Rancher Desktop (having came from Docker Desktop) and I'm running into an issue when using nerdctl to build an image from a Dockerfile. Namely, I get this error (I issue a nerdctl build . command): error: failed to solve: ubuntu18.04 failed to do request: Head "https://registry-1.docker.io/v2/library/ubuntu/manifests/18.04": dial tcp 34.205.13.154443 i/o timeout FATA[0030] unrecognized image format The first line in the Dockerfile is, unsurprisingly, FROM ubuntu:18.04 This built fine in Docker Desktop. What am I missing?

Hi, I used

kube-proxy-arg:- "metrics-bind-address=0.0.0.0"

in config.yml but it seems to still listen on localhost only

tcp        0      0 127.0.0.1:10249

, anyone knows the reason?

Hi does, anybody has an example of terraform code to import generic RKE2 cluster to rancher?

Hi there! Happy Holidays to everyone! I'm setting up a Harvester lab and while trying to solve some issues with volumes provisioning I accidentally deleted the default PV used by Prometheus, thus its pod is always on Pending state as it doesn't seem to automatically recreate. Is there any way to recreate it? I'm thinking on reinstalling the whole Harvester if there's no YAML config available for that

How can I work with package manager in Rancher OS? I am looking to install a package, but there is not apt available. I cannot switch to ubuntu console, as I may face some data loss

Hey, So we are migrating to k8s 1.24 and we use rancher to centralise our cluster (rancher version 2.5.15 using the docker image rancher/rancher) but when connection the cluster I got issue with api , so I had to look into the issue and found out that it might be that rancher versions under 2.6 are not supporting the new k8s version . does anyone faced same issue ? and is it right that v2.5.15 of rancher does not support k8s 1,24 ?

Hi, I enabled etcd in config yaml file like this:

etcd-expose-metrics: true

however etcd is still not exposed, anyone knows what could be the issue?

Good afternoon good people! My team has a vault cluster running on rke2 1.20 but the latest upgrade for vault requires it to run on 1.21+. Since this is a pretty important cluster with stateful sets and all our vault secrets we weren't 100% sure if we could simply upgrade both the rke2 master and agent by joining a new server to the cluster and make the leader election and call it a day or if there is a more preferred way to do this. Any thoughts? My team can provide me with more details as needed if you have follow up questions (and if there is a better channel to ask this, please let me know)

Hi, in our old rancher 2.4.8 version we have "Catalog" section, but in new 2.7.0 I don't see that in a menu, was it replaces to something else?

Hi all, can anyone point me to some docs that detail how to restore an etcd database from a rancher backup? We had a failure of our etcd node and it is no longer connected to the cluster - we have backups on s3 but can't find any details to manually use them

or if there is a way to create a new cluster based on that backup we'd be fine with that as well

h, I can't pull image inside rancehr

Hey all - I’m hoping this is a stupid question. I’m running a POC using a standalone docker installation… very vanilla installation:

community.general.docker_container:
      name: rancher
      image: rancher/rancher:stable
      ports:
      - "80:80"
      - "443:443"
      volumes:
      - /opt/rancher:/var/lib/rancher
      restart_policy: unless-stopped
      privileged: yes

When I try to build a cluster, I get:

failed to create fleet-default/islab-kc01 <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>, Kind=Cluster for rke-cluster fleet-default/islab-kc01: Internal error occurred: failed calling webhook "<http://default.cluster.cluster.x-k8s.io|default.cluster.cluster.x-k8s.io>": failed to call webhook: Post "<https://webhook-service.cattle-system.svc:443/mutate-cluster-x-k8s-io-v1beta1-cluster?timeout=10s>": service "webhook-service" not found

It’s RKE2 and all the cluster settings are default.

Also the kublet has the extra mounts and the iscsi tools are installed, other things are accessing the iscsi portals on the targeted server without issue, its just the new k8s cluster that has problems

hey, anyone can help why the cluster agent disconnected error due to the ca cert issue ? Thx

chardmaster@chardmaster:~$ journalctl -u rancher-system-agent.service

Dec 30 024730 chardmaster systemd[1]: Started Rancher System Agent.

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Rancher System Agent version v0.2.13 (4fa9427) is starting"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Starting remote watch of plans"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Starting /v1, Kind=Secret controller"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Detected first start, force-applying one-time instruction set"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-k3s:v1.24.8-k3s1 to directory /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a86>

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"

Dec 30 024730 chardmaster rancher-system-agent[3478]: time="2022-12-30T024730Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-k3s:v1.24.8-k3s1"

Dec 30 024733 chardmaster rancher-system-agent[3478]: time="2022-12-30T024733Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/installer.sh"

Dec 30 024733 chardmaster rancher-system-agent[3478]: time="2022-12-30T024733Z" level=info msg="Extracting file k3s to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/k3s"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/run.sh"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stderr] + cp -f /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d>

1c8375d7b023a5fe36a4f21bd3135_0stderr] + chmod 755 /usr/local/bin/k3s"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + chown root:root /usr/local/bin/k3s"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + mkdir -p /var/lib/rancher/k3s"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + RESTART_STAMP_FILE=/var/lib/rancher/k3s/restart_stamp"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + [ -f /var/lib/rancher/k3s/restart_stamp ]"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + [ -n efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + [ != efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + FORCE_RESTART=true"

1c8375d7b023a5fe36a4f21bd3135_0stderr] + env INSTALL_K3S_FORCE_RESTART=true INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true installer.sh"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Skipping k3s download and verify"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Skipping installation of SELinux RPM"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kub>

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Creating /usr/local/bin/crictl symlink to k3s"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Creating /usr/local/bin/ctr symlink to k3s"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Creating killall script /usr/local/bin/k3s-killall.sh"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] systemd: Creating service file /etc/systemd/system/k3s.service"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] systemd: Enabling k3s unit"

Dec 30 024738 chardmaster rancher-system-agent[3478]: time="2022-12-30T024738Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stderr] Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/>

Dec 30 024739 chardmaster rancher-system-agent[3478]: time="2022-12-30T024739Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] [INFO] systemd: Starting k3s"

Dec 30 024745 chardmaster rancher-system-agent[3478]: time="2022-12-30T024745Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stderr] + [ -n efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

Dec 30 024745 chardmaster rancher-system-agent[3478]: time="2022-12-30T024745Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stderr] + echo efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4"

Dec 30 024745 chardmaster rancher-system-agent[3478]: time="2022-12-30T024745Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"

Dec 30 024745 chardmaster rancher-system-agent[3478]: time="2022-12-30T024745Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3>

Dec 30 024745 chardmaster rancher-system-agent[3478]: time="2022-12-30T024745Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"

Dec 30 024746 chardmaster rancher-system-agent[3478]: time="2022-12-30T024746Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] Name Location Size Created"

Dec 30 024746 chardmaster rancher-system-agent[3478]: time="2022-12-30T024746Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 024747 chardmaster rancher-system-agent[3478]: time="2022-12-30T024747Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 024752 chardmaster rancher-system-agent[3478]: time="2022-12-30T024752Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 024752 chardmaster rancher-system-agent[3478]: time="2022-12-30T024752Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 025733 chardmaster rancher-system-agent[3478]: time="2022-12-30T025733Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221230-025733/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3>

Dec 30 025733 chardmaster rancher-system-agent[3478]: time="2022-12-30T025733Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"

Dec 30 025733 chardmaster rancher-system-agent[3478]: time="2022-12-30T025733Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0stdout] Name Location Size Created"

Dec 30 025733 chardmaster rancher-system-agent[3478]: time="2022-12-30T025733Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"