sorry that was meant for the thread

Hello y'all. I have upgraded my kubernetes version with an RKE1 template to kubernetes 1.24 and now none of my Persistent Volume Claims will mount with my pods. We are using azure for our PVCs and I made sure the template has the correct azure service principal to be able to connect. Does anyone have any experience with troubleshooting this? The error looks like this:

Failed Mount 
Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[ready-files kube-api-access-8jhkt data-0 app-name-tmp cluster-ca broker-certs client-ca-cert kafka-metrics-and-logging]: timed out waiting for the condition

Hey guys! I did setup for 3 masters and 5 worker nodes It was running well previously but now I am seeing a issue related to etcd not able to connect

k3s[1510]: {"level":"warn","ts":"2023-06-26T04:33:34.490Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000d8a000/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
k3s[1510]: time="2023-06-26T04:33:34Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
k3s[1510]: time="2023-06-26T04:33:34Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"

Tried few solutions from online but nothing helped https://github.com/etcd-io/etcd/issues/12234 https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/07-bootstrapping-etcd.md#configure-the-etcd-server https://stackoverflow.com/questions/55571566/unable-to-bring-up-kubernetes-api-server Can anyone help or tell how to bring etcd service back?? when I try to manually run etcd it starts as expected but k3s service doesnt start because port 2379 is already in use Also port 2379 and 2380 are open on all nodes

Hi Team, I have ran into an issue where we have an application deployed that uses a lot of ingresses. Now I see that rancher puts an annotation on de deployment named

<http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>

at this point the annotation is so big that updates fail with the following error.

is invalid: metadata.annotations: Too long: must have at most 262144 bytes

. Is it possible to disable the use of this annotation for rancher?

Has anyone gotten the UserNamespacesStatelessPodsSupport feature gate to work on either k3s or rke2? I've tried on the most recent versions of both but get the same problem even with the simple example doc'd on the k8s website. (keywords: hostUsers, KEP-127)

Dear all, I have 2 “Machines” in a very weird status: “Failed deleting server [fleet-default/…-cluster-master-pool-c4…] of kind (OpenstackMachine) for machine …-cluster-master-pool-58… in infrastructure provider: DeleteError: About to remove …-cluster-master-pool-c4f…WARNING: This action will delete both local reference and remote instance. (…-cluster-master-pool-c4f…) deleting instance... map[MachineId:] (…-cluster-master-pool-c4…) Deleting OpenStack instance... (…-cluster-master-pool-c4…) Authenticating... map[ApplicationCredentialId: ApplicationCredentialName: AuthUrl:https://X.Y.Z:5000 CaCert: DomainId:53…9 DomainName: Insecure:true TenantDomainID: TenantDomainName: TenantID:2a…c4 TenantName: UserDomainID: UserDomainName: UserId: Username:] Error removing host “…-cluster-master-pool-c4…“: You must provide a password to authenticate” I’m not able to delete them (there are 2 of them) from Rancher UI. They don’t exist in Openstack

Would you have some adivce s on what can be done to completely make them dissapear?

found the solution - again “finalizers” sorry for the noise :)

Hi All, I would like to check if anyone deployed pgbouncer in aks without azure vault

We deploy our Rancher cluster via Terraform, and at one point had to raise a cluster_monitoring_input memory setting. Since moving to Rancher's updated alerting/monitoring system this setting has been deprecated, but at this point we can't find a way to remove it from tf/cluster state (meaning this change persists after we attempt to apply it). Any suggestions?

Creating ELB with openstack. It’s in this state forever. Any guides?

Cross post. Need some help on this one. 🙂 (rke2 cluster)

if I'm using nameSuffix is there a way to prevent that suffix from affecting a persistent volume claim?

I am trying to configure kubernetes-dashboard and it looks like it is tls-only by default. Just so I am not missing something here: I am doing TLS Offloading at my loadbalancer. It does not make sense for me to have the services speak tls with self signed certificates inside the cluster, aight?

Hi All... I am experimenting with Rancher Desktop... I am facing issue with Image pull for newly created images at local... POD is not able to find imeages properly from local repository... Someone help me out regrding same...

Hey there, I'm having an issue similar to this one -

kubectl top nodes

shows

unknown

status for 3/4 nodes.

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary

says it can't find the node, but if I run

kubectl get --raw /api/v1/nodes

, it returns info on the Rancher nodes, not the worker nodes as listed in the top command. What could be going on here?

Hi i have a big problem to pull images in my Rencher v 2.7.5 . I want pull images to deploy one deployment from my registry. My registry es Harbor

Hi. After a quick steer on issues creating new clusters with Rancher. Our Rancher (2.7.3) has been great for a long time. Today, we tried to recreate a cluster that was having issues All the nodes are new, but templated to be identical as the previous cluster nodes that worked fine. But for whatever reason it didn't get past running rancher-system-agent on the first node. I get "error applying plan - see rancher-system-agent logs". The logs grumble about there being no certificates in the /var/lib/rancher... directory, which there aren't. I feel like we hit this in the past, and just fixed it with a Rancher upgrade that was due anyway. But I'd like to understand and be able to troubleshoot this properly. I'd assume all the certs are supposed to be generated and pushed automatically by Rancher as part of the cluster registration scripts. Any tips on where to start looking for issues with this process would be appreciated.

❓ Update a Cluster with delay between nodes Is it possible to delay update between Nodes? I have a Cluster, which hosts Pulsar. The Cluster is provisioned with the Hetzner provider. I have very bad experiences restarting the Nodes to fast, because Pulsar needs to replicate its data. It gets unstable, if a node is drained before the replicationprocess has finished. Thats is why I want to wait 10 minutes, until the next Node gets updated. Is there a way to achieve this delay? how is the procedure in similar situations?

Hi Team, My company has 5000 clusters, one cluster running on each store and looking for a good fleet management solution. I saw rancher has some good product like k3s, fleet, etc. Could you please let me know which product best suits for below requirement? 1. Easy to upgrade k8s version on all cluster at a time in a group 2. Centralized Management 3. Can we attach existing on-prem cluster to rancher?

@astonishing-toddler-94422 has left the channel

Hello super community members: If I want to install EKS Cluster in AWS with version 1.25 and Rancher with rancher-stable chart how to be sure that is going to work I'm checking Rancher - EKS Matrix but still have issue to install rancher there

Hi, please redirect me to the right channel if that's not suitable here. We have installed Rancher on a cluster (version 2.7.5), and when we're trying to connect another cluster we're getting the following error message:

Admission webhook "<http://rancher.cattle.io.globalrolebindings.management.cattle.io|rancher.cattle.io.globalrolebindings.management.cattle.io>" denied the request: user "system:serviceaccount:cattle-system:rancher-infra" (groups=["system:serviceaccounts" "system:serviceaccounts:cattle-system" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["secrets"], Verbs:["create"]} {APIGroups:["<http://catalog.cattle.io|catalog.cattle.io>"], Resources:["clusterrepos"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["catalogs"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["cisbenchmarkversions"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["cisconfigs"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["clusters"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["clustertemplaterevisions"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["features"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["fleetworkspaces"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["globaldnses"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["globaldnsproviders"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["kontainerdrivers"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["multiclusterapps"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["nodedrivers"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["nodetemplates"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["podsecurityadmissionconfigurationtemplates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["podsecuritypolicytemplates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["preferences"], Verbs:["*"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["principals"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rancherusernotifications"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rkeaddons"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rkek8sserviceoptions"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rkek8ssystemimages"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["roletemplates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["settings"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["templates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["templateversions"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://project.cattle.io|project.cattle.io>"], Resources:["sourcecodecredentials"], Verbs:["*"]} {APIGroups:["<http://project.cattle.io|project.cattle.io>"], Resources:["sourcecoderepositories"], Verbs:["*"]} {APIGroups:["<http://provisioning.cattle.io|provisioning.cattle.io>"], Resources:["clusters"], Verbs:["create"]} {APIGroups:["<http://rke-machine-config.cattle.io|rke-machine-config.cattle.io>"], Resources:["*"], Verbs:["create"]}; resolution errors: [[<http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> "cluster-admin" not found, <http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> "system:discovery" not found]]

I can both of the

cluster-admin

and the

system:discoveryy

roles in the target cluster. So what's going on why can't we connect? Technical info: • Rancher is installed on AKS cluster version 1.26 • I tried to connect the following types of clusters: ◦ AKS cluster with no RBAC ◦ AKS cluster with RBAC ◦ Linode cluster with RBAC

Hi I am trying to follow https://ranchermanager.docs.rancher.com/v2.6/pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster When I do I dont understand what goes in

<CHART_REPO>

helm install rancher rancher-<CHART_REPO>/rancher --namespace cattle-system \

--set hostname=master1.ods.vuw.ac.nz--set bootstrapPassword=xxxx

So this failed.

-bash: CHART_REPO: No such file or directory

Hi, is there a way to Create Cluster where the Pools for Master and Worker can come from different Providers, i.e. the Masters from vSpehere provider but the Worker based on Bare metal Elemental? Of course in the same Network Topology.

I had a 3 node downstream RKE cluster with all 3 roles applied to the VMs [etcd,controlplane,worker]. I've since added some worker nodes to the cluster and would like to remove the worker role from the etcd/controlplane node. Is there a way to do this? Or do I have to just remove one of those nodes at a time and re-add them with the proper roles ?

Good Morning Everyone I've just installed Docker, Delighted to explore it in combination with other resources.

We have a bunch of GKE clusters on GCP that are unable to upgrade as Rancher is making calls to an API (/apis/policy/v1beta1/podsecuritypolicies) that's deprecated in K8s version 1.25. I have run the kubectl command under "i. See which PSPs are still active in your cluster:" here[0] (returns nothing) and I have upgraded Rancher with the global.cattle.psp.enabled switch set to false yet Rancher keeps calling the API. Does someone have an idea what exactly it is I'm supposed to do to make it stop calling that API? There is one PSP policy that GKE has previously added that GKE will delete when it upgrades to 1.25 (according to Google I should not try to get rid of it myself). PSP is not active but maybe the existence of that policy is causing Rancher to make those calls? Perhaps I can just force GKE to upgrade and hopefully Rancher will continue to function? [0] https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/pod-security-standards