welcome to the nasty world of finalizers. Good to know that there is also a volumeattachment object.

it’s a good concept, but it gives me headaches 😉

do you use containerd as runtiime? crictl is a nice tool to investigate some of these issues

I’m using k3s, so yeah?

ok. that last sentence in the error message is ok, because you removed Longhorn and also automatiically it’s csi driver

but the lines before that is confusing

Yeah it keeps saying that from time to time. I’m trying to find the resource it’s using to determine that there’s a mounted volume. So far no joy.

there is no easy way to discover this unfortunately, you have to dive into that. but it’s a great learning experience. It gives you insight on how everything is tangled together

perhaps json query makes it a little bit easier. but then you have to know how it is structured for your query

json queries gives me headaches as well 🤣

Here’s a strange one - we have a cluster using rancher/nginx-ingress-controller:nginx-1.2.1-rancher1. Been working great. All of a sudden, we’re seeing horrific ingress times upon first page view (browser is idle for upwards of 90 seconds before data is transferred). After this really slow load time, page responses to the same ingress are really fast. Seems to only be impacting a single cluster right now, other clusters using the same image are fine. And the controller logs are completely empty other than certificate expiration warnings.

Hi all, I'm running the Rancher local cluster (2.6.5) in a 2 node K3S cluster. We had some operating system updates that caused docker to restart on the 2 nodes, now I'm getting "404 page not found" when trying to go to the rancher URL. Does anyone have any suggestions on how I can troubleshoot the local cluster from the command line?

How to provide access to Cloud credentials view access to end users, Rancher is asking to create new cloud credentials while creating new node templates, instead of using the existing ones

Hello, maybe someone here can help me on this one. a former college has installed rancher and the whole kubernetes net for us and i'm pretty new to these things. We have an Application Cluster and an Management cluster and on the management cluster the Certificate did run out. I couldnt rotate the certificate via the UI for this cluster and if i try it with RKE it seems like it also only rotates the certificates from the Application cluster. I dont know where i can find the missing Certificate or where i can renew it.

Hi All, When we tried to deregister few AWS EKS Clusters from rancher, we see a delete node API calls to all the worker nodes of the clusters triggered by user : 'systemserviceaccountcattle-system:cattle ' and was successful. Because of which nodes were deleted from API server (but the intances were still part of AWS auto scaling group) because of which no nodes are availble to the cluster. We just registered the existing AWS EKS Clusters to racnher and we didn't create those clusters using rancher. We are not able to reprodice the issue and it was not happenend earlier, it is just a single occurence happened on 22Jan'23 when we deregister the EKS Cluster from rancher. Can anyway experience the same issue? can you please help with any leads why it might happened? Kubernetes Version in AWS EKS: 1.20 Rancher Version: 2.6.3

image.png

Trying to install Rancher 2.7.1 on top of RKE2 I ran into this problem:

# helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=<http://rancher02.example.com|rancher02.example.com> --set ingress.tls.source=secret --version 2.7.1
Error: INSTALLATION FAILED: chart requires kubeVersion: < 1.25.0-0 which is incompatible with Kubernetes v1.25.6+rke2r1

How comes? Is there a schedule to support Kubernetes 1.25 or 1.26?

Hi people, i need help with one question... is possible disable shell for users in rancher UI ?

Hi all. Which object is responsible for creating the

helm-operation-

pod when I deploy Rancher with a Helm Chart? I know it is not a deployment, a statefulset or a job, so it must be a custom resource

Any idea why a cluster would have BOTH kube-dns and coredns pods running?

Question on node upgrades. We have a completely new node template that we want to use, moving from ROS to Ubuntu. We currently use 3 combination master/etcd nodes. I don’t want to just switch the template and hope it works, I’d rather add new nodes and remove the old. Just trying to figure out how to best swing the etcd and order of operations. If I add 1 new combination etc/master node, that’ll throw etcd off since there would be an even number of nodes. If I add 3 new combination etc/master nodes, that’ll still be an even number. If I add 1 new combination etc/master node while simultaneously scaling down the old pool to 2, maintaining 3 etcd nodes. Scale the old node pool down to 0, increase new node pool to 3. What order will this happen in and will etcd remain happy?

Hello everyone. Is there any info about the cert-manager versions that are supported by rancher 2.7.1? In the doc I could only find mention about the fact that rancher 2.6.4 supports cert-manager 1.6.x and 1.7.1. The current cert-manager is 1.11 . Did anyone tried to run rancher with cert-manager > 1.7.x ? Thank you

I'm running Rancher on k3s and getting a 404 when I try to hit the UI, how can I restart the UI? Is it just redeploying the entire rancher app?

How do you force delete a node that is stuck provisioning, when the backing VM is already delete? I Have 3 nodes that somehow used the wrong machine template stuck: Waiting for VMware Tools to come online...; waiting on node-controller

I'm trying to install RKE2 version "v1.26.1+rke2r1" and it's failing to install canal.

kubectl logs -n kube-system rke2-canal-m2ddv install-cni

says:

2023-01-31 195205.466 [ERROR][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.43.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/canal/token": dial tcp 10.43.0.1

443 i/o timeout

I am using the

<https://get.rke2.io>

installer running on a fresh install of Ubuntu 20.04.4, and the only configuration I have done to is to configure NetworkManager to ignore CNI hosts. How can I resolve this issue?

I'm still having trouble with my Rancher local cluster, 2.6.5 running on rke 1.23. This error is in my rancher pod logs, I'm not sure if it's a red herring or something that could be causing my cluster issues. My k3s cluster seems fine, I get a 404 trying to hit the rancher UI and I'm not quite sure how to troubleshoot the local cluster. I went through the Network Troubleshooting and HA Troubleshooting and everything seems fine, my 3 rancher replicas are running

[ERROR] error syncing 'system-library': handler system-image-upgrade-catalog-controller: upgrade cluster local system service alerting failed: template system-library-rancher-monitoring incompatible with rancher version or cluster's [local] kubernetes version, requeuing

This is a production system, our RKE cluster is still running but who knows for how long, I'm a bit concerned that I can't get to the UI and the local cluster isn't in the best shape

Hi All, Can anyway experience the same issue? can you please help with any leads why it might have happened?

Hi, Getting the following error when tried to run (Cluster Version: 2.3.5)

kubectl auth can-i create pods --as system:serviceaccount:default:spark                                           
Error from server (Forbidden): {"Code":{"Code":"Forbidden","Status":403},"Message":"<http://clusters.management.cattle.io|clusters.management.cattle.io> \"c-w4v2b\" is forbidden: User \"system:serviceaccount:default:spark\" cannot get resource \"clusters\" in API group \"<http://management.cattle.io|management.cattle.io>\" at the cluster scope","Cause":null,"FieldName":""} (post <http://selfsubjectaccessreviews.authorization.k8s.io|selfsubjectaccessreviews.authorization.k8s.io>)

I'm trying to run Rancher Desktop 1.7.0 on Mac 12.6.2(Monterey) and when I open UI, I see error below. I tried to resolve by granting permissions on the dir but issue is not resolved. Is there a solution?

'time="2023-02-01T18:35:37+05:30" level=fatal msg="failed to run [sudo --user root --group wheel --non-interactive /bin/mkdir -m 775 -p /private/var/run]: stdout=\\"\\", stderr=\\"sudo: a password is required\\\\n\\": exit status 1"\n',
  code: 1,
  command: [
    '/Applications/Rancher <http://Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl|Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl>',
    'start',
    '--tty=false',
    '/Users/xxx.xxx/Library/Application Support/rancher-desktop/lima/_config/0.yaml'
  ]
}