#CheckIn Today: • Resolve multifactor authentication for gitlab • Input finished work through Gitlab including security review for VPN on AWS • Get with Ashley regarding bucket policies with IAM policies • Go through to do list tasks on Gitlab

Hi folks, I am trying to recover from an etcd database corruption of a cluster setup by Rancher. There is just a single control plane node and 4 workers. I found https://rancher.com/docs/rke/latest/en/etcd-snapshots/, saying to read the rke documentation for restore. This document recommends to run

# ~/rke.v1.3.15 etcd snapshot-restore --config ~/kube002.config.yaml --name c-blwbg-rl-mxhdw_2022-12-13T05:18:35Z 
INFO[0000] Running RKE version: v1.3.15                 
INFO[0000] Checking if state file is included in snapshot file for [c-blwbg-rl-mxhdw_2022-12-13T05:18:35Z] 
FATA[0000] Cluster must have at least one etcd plane host: please specify one or more etcd in cluster config

Problem is, where do I get a working cluster.yaml from? I had expected that all necessary information is stored in the snapshot (next to rke.state and the etcd database). The cluster.yaml I saved via the GUI appears to be incomplete. Every helpful comment is highly appreciated

I'm trying to configure Rancher Desktop to work behind a proxy. To troubleshoot I have it setup to use mitmproxy. I followed steps here: https://github.com/rancher-sandbox/rancher-desktop/issues/2259#issuecomment-1136833849 I can see that dockerd and curl are correctly using the proxy when I run rdctl shell but I don't see any attempt from the Rancher Desktop client running on the host to use the proxy.

@ancient-bear-48931 Anyone have experience in getting updated container images to address CVEs found during monthly scans?

Hi, everyone! I've deployed sucessfully a k8s cluster in a rancher single node instance, using a custom certificate, but now I'm stucked in a continuous message in the Rancher's panel, that says: Waiting for API to be available anyone could to help me?

so when you remove the default nginx ingress from rancher cluster management, it deletes the entire ingress-nginx namespace.... welp there goes my other nginx controllers i guess 😞

so when i install ranchers logging in cluster tools i get tons of dns errors, looks like the logging chart doesnt support custom cluster domains? Its slamming .svc.cluster.local even with mine not being that domain (staging.cluster.etc.etc)

Is anything special required to use RD on Windows 10 with WSL2? I installed RD, uninstalled DD, restarted wsl2, but am getting this error when running docker inside the wsl2 Ubuntu container

$ docker

Command 'docker' not found, but can be installed with:

sudo apt install docker.io # version 20.10.12-0ubuntu4, or

sudo apt install podman-docker # version 3.4.4+ds1-1ubuntu1

I can't remember if I needed to do

sudo apt install <http://docker.io|docker.io>

when I installed Docker Desktop. Running

docker

in PowerShell seems to be working fine.

Hi all, I have one docker node run rancher and rke cluster. Where is kubeconfig file of rke cluster save at? I want set a cron job to backup same setting that require kubeconfig file, I know WEB UI can get the kubeconfig, but it will change after reboot.

Hello, I have a cluster with 3 masters and 3 workers. When I reboot a master server, it just disappears from the "kubectl get nodes" output. I can't rejoin the master. I can't even add new master to the cluster. Has anyone have this problem, how should I approach it? My RKE2 version is v1.24.8

hello everyone

I got a problem with rancher, suddenly all service can not access, return the 503 error code from nginx

I'm using Rancher Desktop and have some problems running docker on windows. It works great in WSL2 but when I try to run basically any command from PowerShell, I get bellow err. message: I'm running PowerShell as Admin. PS C:\Users\fdrzewieck001> docker config ls error during connect: In the default daemon configuration on Windows, the docker client must be run with elevated privileges to connect.: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/configs": open //./pipe/docker_engine: The system cannot find the file specified. PS C:\Users\fdrzewieck001> docker run hello-world docker: error during connect: In the default daemon configuration on Windows, the docker client must be run with elevated privileges to connect.: Post "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/containers/create": open //./pipe/docker_engine: The system cannot find the file specified.

We are seeing in the UI that Rancher arent always able to to a proper inverntory of the etcd snapshots. And thus we dont get status or size on the actual snapshot in the UI. However checking locally on the control plane nodes we do see that the snapshots has gone through as expected. What can be the cause of this? https://github.com/rancher/rancher/issues/39703

👋 Hi everyone!

I am new to rancher. I have launch a docker container for rancher.. Rancher UI is available but when I am trying to import EKS it shows "Failed to communicate with cluster: Unauthorized" and in the container log it shows "2022/12/14 12:01:40 [ERROR] error syncing 'c-wbp29': handler eks-operator-controller: Unauthorized, requeuing 2022/12/14 12:01:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-wbp29: secrets "cattle-global-data/" not found, requeuing 2022/12/14 12:03:41 [ERROR] error syncing 'c-wbp29': handler eks-operator-controller: Unauthorized, requeuing 2022/12/14 12:03:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-wbp29: secrets "cattle-global-data/" not found, requeuing 2022/12/14 12:04:40 [INFO] checking cluster [c-wbp29] upstream state for changes 2022/12/14 12:04:41 [INFO] cluster [c-wbp29] currently updating, skipping spec sync 2022/12/14 12:04:41 [ERROR] error syncing 'c-wbp29': handler eks-operator-controller: Unauthorized, requeuing 2022/12/14 12:04:41 [ERROR] error syncing 'c-wbp29': handler eks-operator-controller: Unauthorized, requeuing 2022/12/14 12:05:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-wbp29: secrets "cattle-global-data/" not found, requeuing 2022/12/14 12:06:41 [ERROR] error syncing 'c-wbp29': handler eks-operator-controller: Unauthorized, requeuing 2022/12/14 12:07:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-wbp29: secrets "cattle-global-data/" not found, requeuing"

Need some help to resolve this issue...I am stuck for a long and not able to find any solution

I'm wondering the best way to get the kubeconfig passed down to a driver. I was looking at some existing drivers, and saw the harvester node driver accepts the kubeconfig as a flag. Is that avoidable? Using the in-cluster kubeconfig doesn't seem to work, unfortunately.

is there a way to update the status.nodePlan.plan on an old rke node? it is spawning old rancher-agent image which fails after OS upgrade

Hi. I'm trying to get my head around NFS permissions for the external nfs storage class. We've set up the nfs-subdir-external-provisioner and have an NFS server set up and all that good stuff. I can deploy a workload and request a PV to be created and that works. What I'm wondering about is permission handling. Say we have an NFS server that we want to have available to users on Linux systems direct via NFS where people can put files in their PVCs, yet still have it functional in K8s. It seems like when the provisioner creates a volume, it's owned by 'nfsnobody' on our server, with 777 permissions on it. Meaning anyone who can access our nfs server can theoretically access any PVC inside of it. What I'd like ideally is that the user that's submitting a workload 'own' the NFS directory where the PVC resides. Has anyone done anything like this or dealt with closing down NFS permissions for this?

Hello everyone, i'm trying put my AWS Credentials to Rancher Server (this are deployed on Docker Container), but, i'm getting the error: Authentication test failed, please check your credentials. But, i'm already test these credentials on my computer and in EC2, and i receive success in test these credentials. can anybody help me? Really thanks!

message has been deleted

If I install CAPI providers onto a cluster with Rancher installed, what is the interaction? I would like Rancher to not interfere with the CAPI provider resources - i saw rancher has its own internal CAPI controller for RKE2

Hello~ Does anyone knows why 'ImageGCFailed' happen in a cluster using RKE? The kubelet on this cluster has errors with log as below

$ kubectl describe node master2

...
Wanrning ImageGCFailed     5m        kubelet      wanted to free 1000000000 bytes, but freed 0 bytes space with errors in  image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict:
  unable to remove repository reference "<image registry url>/rancher/hyperkube:v1.24.4-rancher1"
  (must force) - container 4b51a3451d24 is using its referenced image 517766856680,
  rpc error: code = Unknown desc = Error response from daemon: conflict: unable to
  remove repository reference "<image registry url>/rancher/mirrored-coreos-etcd:v3.5.4"
  (must force) - container 9ef807f9a241 is using its referenced image 77b8864f9930,
  rpc error: code = Unknown desc = Error response from daemon: conflict: unable to
  remove repository reference "<image registry url>/rancher/rke-tools:v0.1.87"
  (must force) - container 0886469c9f2f is using its referenced image caffe885434d]

Please let me know if you all have resolved error like this.

Hello everyone 🙂, Is there a way/solution to receive rancher metrics in prometheus of ResourceQuotas from a Project? I would like to get Cpu/Memory Limits and Utilization by projects

Hi, how do I revoke a leaked user API Key?

Heya brain-trust. I’m trying to find out workloads on some of our clusters are using API’s that are being removed, from the kubernetes docs https://kubernetes.io/blog/2020/09/03/warnings/#deprecation-warnings i found this handy Prometheus query:

apiserver_requested_deprecated_apis{removed_release="1.22"} * on(group,version,resource,subresource)
group_right() apiserver_request_total

However when i try it i get the following error:

Error executing query: found duplicate series for the match group {group="<http://apiextensions.k8s.io|apiextensions.k8s.io>", resource="customresourcedefinitions", version="v1beta1"} on the left hand-side of the operation: [{__name__="apiserver_requested_deprecated_apis", endpoint="https", group="<http://apiextensions.k8s.io|apiextensions.k8s.io>", instance="172.30.101.215:6443", job="apiserver", namespace="default", removed_release="1.22", resource="customresourcedefinitions", service="kubernetes", version="v1beta1"}, {__name__="apiserver_requested_deprecated_apis", component="k3s-server", endpoint="metrics", group="<http://apiextensions.k8s.io|apiextensions.k8s.io>", instance="172.30.101.215:10250", job="k3s-server", metrics_path="/metrics", namespace="cattle-monitoring-system", pod="pushprox-k3s-server-client-6stpf", removed_release="1.22", resource="customresourcedefinitions", service="pushprox-k3s-server-client", version="v1beta1"}];many-to-many matching not allowed: matching labels must be unique on one side

Being a novice in PromQL i don’t quite get how to resolve this.. any pointers? 🙂

Hello folks! Is it possible to change the font-size in the logs explorer and the shell tabs in the UI?

Regarding https://rancher-users.slack.com/archives/C3ASABBD1/p1671015319219489?thread_ts=1663437408.124819&amp;cid=C3ASABBD1 What about the metrics url https://<rancher-url>/metrics? This returns upto version 2.6.0 only standard go client metrics:

go_**

, `process_*`* and

promhttp_metric_handler_requests_total

Will this be extended in the future?