14

and

30

are not in the cluster.yml, etcd doesn’t report them, and

--etcd-servers

is correctly set to

<https://192.168.253.47:2379>

I’ve rebooted all the machines, so why is

kubectl get no

mis-reporting?

I’m reluctant to add new (replacement) etcd nodes until I understand why

kubectl get no

is misreporting.

Is this better handled as an rke repo issue?

Hi,Im trying to install RKE2 setup. I have created one bootstrap node and trying to add worker nodes. and failed with following error systemctl start rke2-server.service Job for rke2-server.service failed because the control process exited with error code. See "systemctl status rke2-server.service" and "journalctl -xe" for details. [root@rke-master-01 ~]# systemctl status rke2-server.service ● rke2-server.service - Rancher Kubernetes Engine v2 (server) Loaded: loaded (/usr/lib/systemd/system/rke2-server.service; enabled; vendor preset: disabled) Active: activating (start) since Mon 2023-05-15 194022 CST; 1s ago Docs: https://github.com/rancher/rke2#readme Process: 22216 ExecStopPost=/bin/sh -c systemd-cgls /system.slice/%n | grep -Eo '[0-9]+ (containerd|kubelet)' | awk '{print $1}' | xargs -r kill (code=exited, status=0/SUCCESS) Process: 22227 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Process: 22225 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 22222 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS) Main PID: 22228 (rke2) Tasks: 7 Memory: 17.2M CGroup: /system.slice/rke2-server.service └─22228 /usr/bin/rke2 server May 15 194022 rke-master-01 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)... May 15 194022 rke-master-01 sh[22222]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service May 15 194022 rke-master-01 sh[22222]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory May 15 194022 rke-master-01 rke2[22228]: time="2023-05-15T194022+08:00" level=warning msg="not running in CIS mode" May 15 194022 rke-master-01 rke2[22228]: time="2023-05-15T194022+08:00" level=info msg="Applying Pod Security Admission Configuration" May 15 194022 rke-master-01 rke2[22228]: time="2023-05-15T194022+08:00" level=info msg="Starting rke2 v1.25.9+rke2r1 (842d05e64bcbf78552f1db0b32...ea403a0)" Hint: Some lines were ellipsized, use -l to show in full.

Hi there. I’m currently doing some research on two distributions of kubernetes:

rke2

and

k3s

. So far I’ve found these differences between them: • k3s optimized for edge computing, • k3s has diverged from upstream k8s, while rke2 is much closer to upstream • k3s more lightweight, stripped down for edge computing / rke2 is light but not that light • rke2 security-oriented for FIPS compliance • k3s has better community support • i also noticed k3s really starts as a single binary, and controls cluster components internally via it’s own supervisor, but rke2 actually creates static pods for internal cluster components. Not sure which option is better. Our use case is cloud computing, deploying web applications, data pipelines, etc. We don’t really need all that FIPS compliance, but we also run kubernetes on very powerful nodes, so resources aren’t really an issue. We have about 40 nodes in a single cluster right now, but on the new infrastructure we’ll split them across 5 clusters (for geo-rendundancy, plus complete isolation for some specific apps). We’re deploying the clusters via ansible + kustomize + helm (not helm-controller, just helm template + apply resulting manifest after a short diff/review). Given these facts, what distribution would you run? k3s or rke2? Which do you think has a “brighter” future? Thanks ✌️

Do we need (is there) an

rke

channel?

Hi, Is it possible to add extra disks to a machine-pool created using the vsphere cloud provider? We want to install Rook for providing storage in the down stream clusters. But as the prerequisistes state, we can only use one of the following local storage types: • Raw devices (no partitions or formatted filesystems) • Raw partitions (no formatted filesystem) • LVM Logical Volumes (no formatted filesystem) The persistent volume using a storage class is not a valid option for us at this moment.

Hi 👋 I run in a real world example about why we run multiple instances of control plane and etcd: • I run rancher 2.5.15 with 2 controlplane and 3 etcd (and a dozen of big worker nodes) • One node with a controlplane and etcd get unstable and I planned its replacement • While setup the next node, the node owning the third etcd copy just blown (disk crashed) So right now, the cluster has the current status: • for some reason rancher try to rotate certificates • the node with controlplane and etcd try to get elected leader, but can't reach the unstable node (reboot kube containers in loop) to validate the election Is there a way to stop the in progress cert rotation? Should I remove the unstable node from the cluster (so the remaning controlplane and etcd can get elected since there won't be any other candidates) ? Thank you

Hi i rancher v2.6.6 and have helm chart and i put it in the "Cluster managment repositories" i want it to be in all the clusters "Apps" > Repositories/Charts How can i configure it?

Hi! In the latest rancher v2.7.3 rancher webhook is also deployed in the downstream cluster. How can I redeploy it in case it is accidentaly deleted? Unfortunately I followed wrong KB https://www.suse.com/support/kb/doc/?id=000020699 😞 . Thanks in advance!

Hi all. We run Rancher 2.6.11 and have 4 downstream clusters. All these clusters run Ubuntu 22.04.x with docker 20.10.24 Upgrading from 1.23.x to 1.24.x we've seen several issues that we can't pinpoint yet to the root cause. 1. When upgrading the Kubernetes version from 1.23.x to 1.24.x All containers were restarted. We assumed this was related to the docker-cri change. However when we upgraded from 1.24.10 to 1.24.13 we saw the same behaviour. 2. In the upgrade process it seems to lose the tunnel a lot of the times with the following error:

[ERROR] Failed to set up SSH tunneling for host [10.0.10.107]: Can't retrieve Docker Info: error during connect: Get "<http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info>": can not build dialer to [c-p9cqp:m-2bd0a9380d15]
[ERROR] Removing host [10.0.10.107] from node lists

A restart of the affected node is required to recover it. The restart of all containers is very worrying and it's something I have never seen before. I've tried to search about the behaviour but no luck so far. Has anyone seen this behaviour before?

Hello, I'm new here, but I would like some help with my EKS cluster on Rancher 2.5.5 (i know it's depreciated, but we couldn't update yet). Well, I just can't access my cluster, it keeps saying "*Error* cannot determine access, cluster is unavailable". I've tried with admin root user, tried to create another admin user, but none worked. All other clusters on this rancher are working fine, just this one has this error and I don't know what to do. Important to note that last friday we had this access working, but today we do not. No one worked on the weekend, so we do not had any changes between friday and today. I have the access using AWS Management Console, but I need to access using rancher, not EKS console. Any idea what's going on here?

Hello ! Is it possible to set up Rancher to use Let's Encrypt as a Certificate Issuer using DNS challenge for authentication?

v1beta1.custom.metrics.k8s.iocattle-monitoring-system/rancher-monitoring-prometheus-adapterFalse (FailedDiscoveryCheck)

pls help

Hey here 👋 Is there a way to stop an infinite certificate rotation? (some nodes are not reachable right now and it rotates every 15min w/o stopping)

What happened in this dashboard before upgrading to 2.7.3? It is showing up pods information here, btw I'm using EKS 1.25

Is there a timeline available when to expect the next rancher release?

Hi, Can we use Rancher desktop to manage multi cloud clusters as well or manage only local environment clusters?

Been trying to read up on everything I can find but still not closer to finding a reason or solution to https://rancher-users.slack.com/archives/C3ASABBD1/p1684163104935889 The support matrix indicates everything we run should be supported ( Ubuntu and docker versions ) so still not much luck in finding a solution. The main concern is that all containers are being restarted which is something I've never seen happen before.

Hello guys, currently the kubernetes cluster gets configured with an ip address which interferes with a machine in our network. Is it possible to change the subnet over which the ip addresses are generated?

Hi, I have my cluster in a broken state with

AgentDeployed False

and it seems that my control plane is broken, the thing is, that I cant create more control plane nodes because they get stuck with

Waiting for registering with kubernetes

, Rancher version is 2.7.3 and Kubernetes version is 1.24.13 SSHing into those machines I just see 2 containers with

docker ps -a

CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS                      PORTS     NAMES
ed435976827f   rancher/rancher-agent:v2.7.3   "run.sh --no-registe…"   10 minutes ago   Exited (0) 10 minutes ago             share-mnt
44860d74081d   rancher/rancher-agent:v2.7.3   "run.sh --server htt…"   10 minutes ago   Up 10 minutes                         admiring_yalow

Taking a look into the logs of the

rancher-agent

that it's still running I get

INFO: Arguments: --server <https://REDACTED> --token REDACTED -r -n m-644rr
INFO: Environment: CATTLE_ADDRESS=172.31.37.252 CATTLE_AGENT_CONNECT=true CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME=m-644rr CATTLE_SERVER=<https://REDACTED> CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver 127.0.0.53 options edns0 trust-ad search us-east-2.compute.internal
WARN: Loopback address found in /etc/resolv.conf, please refer to the documentation how to configure your cluster to resolve DNS properly
INFO: <https://REDACTED/ping> is accessible
INFO: REDACTED resolves to REDACTED
time="2023-05-16T20:33:41Z" level=info msg="Listening on /tmp/log.sock"
time="2023-05-16T20:33:41Z" level=info msg="Rancher agent version v2.7.3 is starting"
time="2023-05-16T20:33:41Z" level=info msg="Option worker=false"
time="2023-05-16T20:33:41Z" level=info msg="Option requestedHostname=m-644rr"
time="2023-05-16T20:33:41Z" level=info msg="Option dockerInfo={PFIV:LBRE:AMMF:JNWU:4XQZ:GPJN:4FSD:4O6A:U336:VFBT:7WVD:AKOS 2 1 0 1 1 overlay2 [[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] [] {[local] [bridge host ipvlan macvlan null overlay] [] [awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} true true false false true true true true true true true true false 32 false 39 2023-05-16T20:33:41.064129161Z json-file systemd 2 0 5.19.0-1025-aws Ubuntu 22.04.2 LTS 22.04 linux x86_64 <https://index.docker.io/v1/> 0xc0011f0a10 4 16629444608 [] /var/lib/docker    control-plane-5 [provider=amazonec2] false 20.10.23   map[io.containerd.runc.v2:{runc [] <nil>} io.containerd.runtime.v1.linux:{runc [] <nil>} runc:{runc [] <nil>}] runc {  inactive false  [] 0 0 <nil> []} false  docker-init {3dce8eb055cbb6872793272b4f20ed16117344f8 3dce8eb055cbb6872793272b4f20ed16117344f8} {v1.1.7-0-g860f061 v1.1.7-0-g860f061} {de40ad0 de40ad0} [name=apparmor name=seccomp,profile=default name=cgroupns]  [] []}"
time="2023-05-16T20:33:41Z" level=info msg="Option customConfig=map[address:172.31.37.252 internalAddress: label:map[] roles:[] taints:[]]"
time="2023-05-16T20:33:41Z" level=info msg="Option etcd=false"
time="2023-05-16T20:33:41Z" level=info msg="Option controlPlane=false"
time="2023-05-16T20:33:41Z" level=info msg="Connecting to <wss://REDACTED/v3/connect> with token starting with c5brthtz9nwjwnmrqr5spckpw45"
time="2023-05-16T20:33:41Z" level=info msg="Connecting to proxy" url="<wss://REDACTED/v3/connect>"
time="2023-05-16T20:33:41Z" level=info msg="Requesting kubelet certificate regeneration"
time="2023-05-16T20:33:41Z" level=info msg="Starting plan monitor, checking every 120 seconds"
time="2023-05-16T20:35:41Z" level=info msg="Requesting kubelet certificate regeneration"

Any ideas why they get stuck at

Requesting kubelet certificate regeneration

?

Hi is there a way to pass $RKE2_TOKEN to get.rke2.io? Or I need to update config before starting the server node? Thanks

Greetings all

I was wondering if anyone could give me a quick pointer. Been struggling with Rancher 2.7, fresh Debian 11 install, everything seems to function ok on the surface. Could not get nodePorts to work, pretty sure the selector/label setup is correct, then trying to diagnose around kube-proxy. Not 100% sure I understand correctly, but don't see a kube-proxy pod in the kube-system namespace, so assume this could be related

Hello, i want ask about "profile: cis-1.23" in RKE2. I installed cluster 1.25 without this option, then I turned on this option and it stopped working: "deployment.apps/rancher-webhook". Shouldn't I use rancher-webhook?

Morning all. Is there a way to disable cert validation when creating cluster with rancher? I use a self-signed cert on the my local Nutanix server and when connection to the cluster to create new nodes, it appears to fail. I believe this is due to the cert. Any ideas?

Greetings all,

I did wrong backup & restore script for rke2 cluster with 3 masters and 2 workers. Step 1: I stopped all 3 master servers Step 2: I execute the command rke2 server\ --cluster-reset \ --cluster-reset-restore-path=<PATH-TO-SNAPSHOT> Step 3: I reviewed the documentation using the command rke2 server --cluster-reset but the process failed due to an error of etcd

{"level":"info","ts":"2023-05-17T08:07:29.022Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--config-file=/var/lib/rancher/rke2/server/db/etcd/config"]}

{"level":"warn","ts":"2023-05-17T08:07:29.022Z","caller":"etcdmain/etcd.go:75","msg":"failed to verify flags","error":"open /var/lib/rancher/rke2/server/db/etcd/config: permission denied"}

Step 4: I restarted rke2-server on 3 master servers and got async error Step 5: On first master execute rke2-uninstall Step 6: I removed etcd on master nodes to start the etcd cluster Step 7: I tried to remove nodes first master to reconfigure join but failed. (kubectl delete nodes master-01) Can anyone help me get the first master into the cluster?