h, I can't pull image inside rancehr

Hey all - I’m hoping this is a stupid question. I’m running a POC using a standalone docker installation… very vanilla installation:

community.general.docker_container:
      name: rancher
      image: rancher/rancher:stable
      ports:
      - "80:80"
      - "443:443"
      volumes:
      - /opt/rancher:/var/lib/rancher
      restart_policy: unless-stopped
      privileged: yes

When I try to build a cluster, I get:

failed to create fleet-default/islab-kc01 <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>, Kind=Cluster for rke-cluster fleet-default/islab-kc01: Internal error occurred: failed calling webhook "<http://default.cluster.cluster.x-k8s.io|default.cluster.cluster.x-k8s.io>": failed to call webhook: Post "<https://webhook-service.cattle-system.svc:443/mutate-cluster-x-k8s-io-v1beta1-cluster?timeout=10s>": service "webhook-service" not found

It’s RKE2 and all the cluster settings are default.

Also the kublet has the extra mounts and the iscsi tools are installed, other things are accessing the iscsi portals on the targeted server without issue, its just the new k8s cluster that has problems

hey, anyone can help why the cluster agent disconnected error due to the ca cert issue ? Thx

chardmaster@chardmaster:~$ journalctl -u rancher-system-agent.service

Dec 30 02:47:30 chardmaster systemd[1]: Started Rancher System Agent.

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Rancher System Agent version v0.2.13 (4fa9427) is starting"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Starting remote watch of plans"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Starting /v1, Kind=Secret controller"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Detected first start, force-applying one-time instruction set"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-k3s:v1.24.8-k3s1 to directory /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a86>

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-k3s:v1.24.8-k3s1"

Dec 30 02:47:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:33Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/installer.sh"

Dec 30 02:47:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:33Z" level=info msg="Extracting file k3s to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/k3s"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/run.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + cp -f /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d>

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + chmod 755 /usr/local/bin/k3s"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + chown root:root /usr/local/bin/k3s"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + mkdir -p /var/lib/rancher/k3s"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + RESTART_STAMP_FILE=/var/lib/rancher/k3s/restart_stamp"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ -f /var/lib/rancher/k3s/restart_stamp ]"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ -n efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ != efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + FORCE_RESTART=true"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + env INSTALL_K3S_FORCE_RESTART=true INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true installer.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Skipping k3s download and verify"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Skipping installation of SELinux RPM"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kub>

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating /usr/local/bin/crictl symlink to k3s"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating /usr/local/bin/ctr symlink to k3s"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating killall script /usr/local/bin/k3s-killall.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] systemd: Creating service file /etc/systemd/system/k3s.service"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] systemd: Enabling k3s unit"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/>

Dec 30 02:47:39 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:39Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] systemd: Starting k3s"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ -n efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + echo efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3>

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"

Dec 30 02:47:46 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:46Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: Name Location Size Created"

Dec 30 02:47:46 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:46Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:52 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:52Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:52 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:52Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221230-025733/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3>

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: Name Location Size Created"

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

Hi, can someone confirm me how I should set the Data Center to the downstream-cluster options under add-ons when using cloud provider vSphere? should it be in format of “/datacentername/cluster” or just “datacentername” or “/datacentername”?

Hi everyone, am on M1 and able to connect to docker. But i am unable to do it via testcontainers I get Could not find a valid Docker environment, anyone faced this issue before

Is there a dark mode for Rancher Desktop?

Hi all

@here … i am using rancher desktop on a M1 mac and v activated kubernetes + traefik + install rancher kubernetes ui …… on the M1 i can visit the rancher page (https://rancher.my.local.domain - private DNS-Server and hosts-File on M1 configured, without entry in hosts file i can’t reach the url locally too) and can do something else with the local cluster ….. but if i try to add a remote cluster (5 node’s k8 cluster (all Ubuntu machines) in my lab, the cattle cluster agent will spawn but cannot connect back to rancher on M1 …. following error in pod-log:

INFO: Environment: CATTLE_ADDRESS=192.168.125.212 CATTLE_CA_CHECKSUM=b613734ff8828536815451b443d012a82e583af53b50291f4940f6bdf32a8204 CATTLE_CLUSTER=true CATTLE_CLUSTER_AG │
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local nameserver 10.96.0.10 options ndots:5                                       │
ERROR: <https://rancher.my.local.domain/ping> is not accessible (Could not resolve host: rancher.my.local.domain)

if i install rancher directly (via helm) on top of the “external” cluster (no rancher on M1) i can reach rancher ui from anywhere and play with the “external” k8-cluster (funny thing: here, i dont neet to configure hosts-Files, only the A record in the DNS-Server ) Any idea whats going wrong or i v to configure ?

@icy-magazine-59042 -> #rancher-desktop

Hi team, I have installed the rancher desktop and i am getting the following error 2023-01-02T11:45:59.093Z: Error starting up: Error: Platform "win32" is not supported by RcFilePathManager at new b (C:\Users\DSPRAKAS\AppData\Local\Programs\Rancher Desktop\resources\app.asar\dist\app\background.js1️⃣19466) at k (C:\Users\DSPRAKAS\AppData\Local\Programs\Rancher Desktop\resources\app.asar\dist\app\background.js1️⃣20819) at C:\Users\DSPRAKAS\AppData\Local\Programs\Rancher Desktop\resources\app.asar\dist\app\background.js:29:96630

Hello rancher-people, I've got a silly problem while using rancher rke2. I added a node (it's a single-node cluster) and i renamed the vm (the node) using hostnamectl. Now, the cluster is not available anymore via rancher-ui. The secrets, the config-maps and other settings are not available anymore. Anybody how to recover it?

Howdy all, running Rancher v2.4.15 in HA with downstream RKE clusters at K8s v1.18.20 and Docker v19.3.15. I’m trying to pull down images from

<http://quay.io|quay.io>

today and I’m hitting this:

rpc error: code = Unknown desc = x509: certificate has expired or is not yet valid

I’ve manually rotated the cluster certs in the Rancher UI for each cluster, and no-go. Also respun the cattle-node-agent and canal pods.

I’m using ROS for my nodes, might I be hitting this: https://github.com/rancher/os/issues/3062 ?

Hi. I am using rancher desktop on a mac M1.

@fast-piano-59234 Hello, - I have accidentally deleted the "Kube-Controller-Manager" docker container of my running RKE cluster. - I thought, it will come up automatically, as it might be running at static pod, but it did not started. Could you please let me know, how can I start "kube-controller-manager" docker container of RKE cluster? Cluster details: - 2 Nodes cluster (Rancher Manager cluster/Upstream) - Both nodes have CP/ETCD/Worker roles - RKE version - 1.22.9

Hi! Would it be possible for you to switch the context? (Click right on the Rancher Desktop tray icon -> Kubernetes Contexts -> Select rancher-desktop)

hello! Can someone help me with a problem ? Im trying to use rancher, firstly worked for me, but today it didnt, i reinstalled it, and now writting the command docker ps i receive "error during connect: In the default daemon configuration on Windows, the docker client must be run with elevated privileges to connect.: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/containers/json": open //./pipe/docker_engine: The system cannot find the file specified." . Also, when im running the diagnostics, i can see this problem : Kubernetes is using context

testing

instead of

rancher-desktop

.

Hi, has anyone else had an outdated rancher app repo? Specifically the "Rancher"-repo (https://git.rancher.io/charts) on just one of my clusters is stuck on "release-v2.6"-branch after upgrading rancher to 2.7.0. All other clusters got the updated branch, release-v2.7. All clusters are k3s, on bare-metal, then imported into rancher

I need an info about the Rancher: Can Rancher considered to be CI CD tool for simple app deployment or its just CD ? Ref links please to understand it

Hi, does anybody had issues with installing bundled packages (istio) in air-gapped env on Rancher 2.7.0?

New to rancher, containers won't run, get this error in rancher/rancher:latest exiting because of error: log: cannot create log: open /tmp/rancher.7b2e51e2ebab.root.log.INFO.20230103-160044.6: no space left on device and exiting because of error: log: cannot create log: open /tmp/rancher.7b2e51e2ebab.root.log.INFO.20230103-160044.6: no space left on device. I dont know how to create space

Good morning! My team has a vault cluster running on rke2 1.20 but the latest upgrade for vault requires it to run on 1.21+. Since this is a pretty important cluster with stateful sets and all our vault secrets we weren't 100% sure if we could simply upgrade both the rke2 master and agent by joining a new server to the cluster and make the leader election and call it a day or if there is a more preferred way to do this. Any thoughts? My team can provide me with more details as needed if you have follow up questions (and if there is a better channel to ask this, please let me know)

when i ssh into the rancher container i get Filesystem Size Used Avail Use% Mounted on /dev/mapper/docker-253:8-125834138-d151ae3d32e20fe3b95d29de0c27ea5f1640c23e95b2cf52c4e50c4385f518d9 10G 10G 56K 100% / tmpfs 64M 0 64M 0% /dev tmpfs 126G 0 126G 0% /sys/fs/cgroup shm 64M 0 64M 0% /dev/shm /dev/mapper/OS_Vol-lv_var 224G 32G 192G 15% /etc/hosts /dev/mapper/OS_Vol-lv_root 10G 4.0G 6.1G 40% /var/lib/rancher tmpfs 126G 0 126G 0% /proc/acpi tmpfs 126G 0 126G 0% /proc/scsi tmpfs 126G 0 126G 0% /sys/firmware

Hey all. I'm having an odd issue with Rancher 2.7.0 and some of our managed clusters. In some clusters, attempting to upgrade an app installed via Helm using the chart repositories under the

Apps

section in the sidebar is failing. When I select

Edit/upgrade

for the already-installed application, I get a blank white screen. Looking in the Chrome developer tools console, I also see a message that says

Find: [management] provisioning.cattle.io.cluster undefined

, followed by a warning that says

Attempting to load a resource with no id

followed by a large stack trace. The cluster in question is an AWS EKS cluster running v1.20.15-eks-99076b2, but I also had this issue on another non-cloud-hosted cluster that was running v1.24.8+k3s1 and that is managed by a different Rancher instance. On that last non-cloud instance, I was able to unenroll and reenroll the cluster with Rancher, which solved the issue. I can provide any other information as necessary.

Hello i am new here :)

hi, is it possible to change general Rancher set up to bundled (_CATTLE_SYSTEM_CATALOG=bundled_) in air-gapped env without recreation Rancher