We need to run etcd defrag on our production rancher cluster, does this cause any downtime? The rancher docs dont say

Is there a way to deploy raw kubernetes manifests without using the UI? I am trying to find a good tutorial on how I can deploy a deployment on a specific cluster with node labels.

Anyone facing this issue https://github.com/rancher/rancher/issues/36868. Installed K3s standalone 1.25 and creating RKE2 cluster 1.24 with 1 master node (all) and 2 worker nodes

Hi Team I installed Nginx Ingress Controller on Rancher k8s through the command line ..now RKE-based k8s don't have a separate service for Nginx Ingress ...like NodePort or load balancer which I can see after the installation ..by default nginx-Ingress is installed as Deamonset ... now to access the application from an outside cluster in the browser I don't have any service controller so that I can use as a load balancer or NodePort.normally Ingress Controller works as a router ... now how can i access my application from outside

Hello guys, we are running ..

VERSION: v1.22.4

For

kind: Service

which is

type: LoadBalancer

we wanted to set

externalTrafficPolicy: Local

so that we only have one registered target under AWS TargetGroup where the actual pod is running.

^^^ Incomplete here is the full info… we are running ..

VERSION: v1.22.4
ON AWS
OS: Amazon Linux 2   4.14 KERNEL
<docker://20.10.4>

For

kind: Service

which is

type: LoadBalancer

we wanted to set

externalTrafficPolicy: Local

so that we only have one registered target under AWS TargetGroup where the actual pod is running. we figured that the name of the node should marry up with what kube-proxy thinks the hostname is. in our case when we run kubectl get nodes we have NAME like - ip-10-103-149-15.eu-west-2.compute.internal and the actual hostname is different (as per below snapshot) in order the update the kube-proxy hostname-override we are changing cluster yaml like below.

kube-proxy:
     extra_args:
        hostname-override: "$(curl <http://169.254.169.254/latest/meta-data/hostname>)"

however we are not able to use hostname-override option by editing cluster.yaml what we have tested is that if we delete the kube-proxy container and run it as below everything works fine -

docker run --net=host --privileged -v /etc/kubernetes:/etc/kubernetes -v /lib/modules:/lib/modules -v /var/lib/docker/volumes/b9be15df5114090439353e899bbe66bff17e7ebffe979edaaebf8e9055703f/_data:/opt/rke-tools -v /run:/run --name kube-proxy --entrypoint "/opt/rke-tools/entrypoint.sh" rancher/hyperkube:v1.22.4-rancher1 kube-proxy --cluster-cidr=10.70.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml --healthz-bind-address=127.0.0.1 --v=2 --bind-address=10.108.151.7 --hostname-override=ip-10-103-149-15.eu-west-2.compute.internal

pls note the

--hostname-verride=_*ip-10-103-149-15.eu-west-2.compute.internal*_

however RKE updates the kube-proxy after sometime, and we are not sure from where we can update it.

Hi guys , i was doing some research on the downstream cluster provisioning via rancher , where i was looking at the differences between the rke and rke2 , and i found the below lines in the article

Cluster API

RKE2 is built on the Cluster API framework, which affects the way cluster configurations are managed and changes are made. Unlike RKE1, RKE2 clusters are more susceptible to node reprovisioning when changes are made to the cluster configuration. This behavior is controlled by the CAPI controllers and not by Rancher itself. Certain changes, such as enabling drain before delete or scaling down the number of nodes, may result in the deletion of existing nodes and the creation of new ones. This is a known issue in the Cluster API that will be fixed in a future release, and Rancher will update the documentation accordingly.

I just want to know whether this issue is resolved in 2.7.4 ?? or is there any workaround to prevent the reprovisioning? Any suggestions and the guidance will be appreciated.

Question about RBAC in Kubernetes: Are access roles cumulative? If a user has access rights A and the posix group the user is in (LDAP) has access rights B, does it end up with the combined access rights A+B? Does Rancher consider the whole set of groups the user is in?

Howdy, How can I test the email setup of a reciever in Rancher monitoring v2 (Rancher 2.6.9)?

I'm reading in the rancher monitoring documentation "_Rancher allows any users who are authenticated by Kubernetes and have access the Grafana service deployed by the Rancher Monitoring chart to access Grafana via the Rancher Dashboard UI_" and am struggling to figure out how can access to grafana service be assigned to users (via their groups).

Hi. Trying to use longhorn CSI snapshot to take backup of PVC. Getting this error "Unable to create snapshots. Unable to get VolumeSnapshotClassList: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io)"

Question about

rancher/system-upgrade-controller

, please feel free to redirect me if I'm in the wrong place 🙂 I'm writing a mutating admission controller to dynamically increase the ActiveDeadlineSeconds for specific batchv1.Jobs submitted by the system-upgrade-controller beyond what is set in system-upgrade-controller's chart's configmap. I think what I'm experiencing though is after I mutate the create request, system-upgrade-controller updates the batch.Job and it resets to the ActiveDeadlineSeconds to what the configmap is set to. At least, that's my primary suspect of where the change is coming from since the kubernetes audit logs show a bunch of PATCH requests from the system-upgrade-controller to the batchv1.Job resource. I updated my mutator to also act on Update events and in doing so I'm able to force the ActiveDeadlineSeconds to be what I want but it doesn't seem like a harmonious approach on my end to have the mutator wrestle against the system-upgrade-controller like this. I'd appreciate any advice on how I can improve this approach.

Is there a way to specify the version of Helm used internally by RKE v1.24.11 to deploy charts using the HelmController, like you can specify the version of kubectl in fleet-agent using global.kubectl.tag? My problem is that when I deploy ArgoCD using a HelmChart CRD I get the following error message in the controller logs: chart requires kubeVersion: >=1.23.0-0 which is incompatible with Kubernetes v1.20.0 which means that the chart needs to be templated into manifests using a version of Helm > v1.20.0. But since I'm running RKE2 v1.24.11 I figured the built-in controller would use a version of Helm new enough to work.

I haven't used my rancher instance since 2.7.1 which was kinda rough for my configuration so I just kinda left it sitting there. I decided to check if there was a new version yesterday and was really surprised to see multiple versions released in the past 90 days ! Was able to upgrade easily to the latest without incident. I wanted to share that it's been a very positive experience. It feels like y'all rewrote the reconciler loops to better handle everything... Things are faster and smarter it seems. Thanks for building a cool product!

is there documentation for rancher 2.6.6 API ? (in developer mode i can see that he send a lot/to much information in the body and it does not work as well.) need api to update a helm charts from "Apps > installed Apps"

Hi, I have a problem exposing my app via ingress and external load balancer (rke2). Setup: • Rancher v2.7.2 • External lvl4 load balancer with SSL termination • Ingress (class NGINX), that connects to my app and URL http://www.example.com/app • App has a ClusterIP I can reach the app from one of the nodes using the ClusterIP and HTTP. However, behind the external load balancer I get an "Bad Gateway". My assumption is that the cluster still gives me an HTTPS header back. I already deactivated this for the Rancher RKE2 cluster but have a hunch that the second "user cluster" that is managed by Rancher and that the app is on ignores that. I tried setting the "use-forwarded-headers: true" option in the cluster YAML but 1. there is no ingress section 2. When I create the ingress section any change is ignored, i.e., when I re-open the YAML the changes are gone Any help/comment highly appreciated

Hi, my Rancher v2.7.1 (deployed in k3s) has just started returning "service unavailable" When I look at the rancher pods they are running, but show as unhealthy. The logs seem to get stuck after "Applying CRD machinesets.cluster.x-k8s.io". When I enable --trace they get stuck at "No active connection for cluster". Rancher is managing the cluster in which its deployed, and two clusters which have our build agents running in them. I think it's the two other clusters that are showing this. We changed nothing on rancher, it just stopped working this morning. Where would be best to get help with this?

Same with 2.7.4, just updated the pods

it's stuck after "Starting cluster controllers for local" and never seems to bind

Whenever I create a cluster using rancher on harvester, all of the VM's have dynamic IP's. Currently, I've deployed another VM with a static IP running HAProxy and a custom script that uses

kubectl get nodes

to generate the HAProxy config, so that my DNS entries can point to a static IP. Is there a better way to do this that I'm missing?

we upgraded some things, that didn't help, then it just started working again...

How does one can reach a proxy url in rancher example :

<https://rancher.prod.ops.example.net/k8s/clusters/c-kasjdfi/api/v1/namespaces/prometheus/services/http:prometheus-kube-prometheus-prometheus:9090/proxy/#/dashboard>

I tried (via grafana ui) using a bearer token and also basic auth without success. How does one authenticate to reach that ?

did anyone encounter the error? : 1 node(s) didn't have free ports for the requested i installed rancher on 3 nodes and install app jfrog docker artifactory

image.png

I need to recover a rancher cluster (using docker setup). I restored from the backup:

docker run  --volumes-from goofy_sinoussi -v $PWD:/backup busybox sh -c "rm /var/lib/rancher/* -rf  && tar pzxvf /backup/rancher-data-backup-v2.7.4-2023-06-19.tar.gz"

Rancher ist starting, I can Login, the cluster shows up but "unavailable" and "Cluster agent is not connected". I know that there is no agent connected, how can i connect an agent ?

HI, I'm trying to create an ingress and am wondering where it runs? When I create one under "ServiceDiscovery->Ingresses->Create" I can't find a pod on any of the nodes that runs this specific ingress. Or does the nginx ingress connect to the rke2 ingress controller? I think I'm still missing something. Maybe another (related) question: how do I create an Ingress Controller in Rancher?

Hi! i've updated to rancher desktop 1.9.0, if i pull docker images, it returns me: docker pull localstack/localstack Using default tag: latest Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) i don't have any network problem, nor space on disk for images....any suggestions?

did anyone managed to install and use rancher on perm using multiple nodes? i keep struggling to achieve stability, almost everything i try to do , app instill (charts,) adding nodes after app installed is very unstable, and requirs custom configurations and tweaks, noting works out of the box

Hi Team, When i am running

flytectl demo start

i am getting following error :

🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
delete existing sandbox cluster [y/n]: y
🐋 Going to use Flyte v1.7.0 release with image <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84>
🐋 pulling docker image for release <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-1ae254f8683699b68ecddc89d775fc5d39cc3d84>
🧑‍🏭 booting Flyte-sandbox container
😬 Something went wrong: Failed to start Sandbox container 🐋, Please check your docker client and try again.
Error: Error response from daemon: driver failed programming external connectivity on endpoint flyte-sandbox (b4e1199dc143e2a67410bf4ea35b1c75dc256cc21c3862833ffb3cb6cf36b179): Error starting userland proxy: listen tcp4 0.0.0.0:6443: bind: address already in use

But when i run the command with option

n

i get following message :

🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
delete existing sandbox cluster [y/n]: n
Existing details of your sandbox👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉
❇️ Run the following command to export sandbox environment variables for accessing flytectl
	export FLYTECTL_CONFIG=/Users/vipul.goswami@schibsted.com/.flyte/config-sandbox.yaml
👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30080/console> 🚀 🚀 🎉
❇️ Run the following command to export demo environment variables for accessing flytectl
	export FLYTECTL_CONFIG=/Users/vipul.goswami@schibsted.com/.flyte/config-sandbox.yaml
🐋 Flyte sandbox ships with a Docker registry. Tag and push custom workflow images to localhost:30000
📂 The Minio API is hosted on localhost:30002. Use <http://localhost:30080/minio/login> for Minio console

But none of the links are working here. Also when running following command gives following error:

pyflyte run --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
	details: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Failed to connect to remote host: Connection refused
	Debug string UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:30080: Failed to connect to remote host: Connection refused {created_time:"2023-06-20T15:46:32.394087+02:00", grpc_status:14}

This is the issue with rancher as when i run the these same command with docker-desktop it work fine so could you guys please help whats the issue under the hood