Hello Everyone. I am trying to restart a Control Plane node. We currently have 2 control nodes running and one deleted. There seems to be an etcd issue and it wont start up whenever we spin up a new node. Docker runs "rancher/rancher-agent:v2.6.13 "run.sh --server htt…"" this container but it gets stuck "Requesting kubelet certificate regeneration". Our cluster is stuck in Updating state with the error: "waiting for control-1 to finish provisioning." We are able to see that our

kube-controller-manager

is set to that node and is currently deleted. Is there anyway to change the kube-controller-manager or get etcd issue to fix so we can redeploy this node? Basically our cluster is stuck waiting to provision a node that doesn't exist, and when we try to create it, it fails.

$ kubectl describe endpoints kube-controller-manager  -n kube-system
Name:         kube-controller-manager
Namespace:    kube-system
Labels:       <none>
Annotations:  <http://control-plane.alpha.kubernetes.io/leader|control-plane.alpha.kubernetes.io/leader>:
                {"holderIdentity":"control-1","leaseDurationSeconds":15,"acquireTime":"2022-01-11T16:00:56Z...
Subsets:
Events:  <none>

Does anyone have any clue on how to debug my above post? Or how to get our new control node back up? Would really appreciate any help.

Is there any way to configure Garbage Collection for images and containers via kubelet settings in RKE1?

Couldn’t find it in the docs

Hi! How come RKE1 has 1.3.x and 1.4.x releases concurrently? Is there a reason not to use 1.4.x (coming from 1.3.x)? I don’t see anything in the 1.4.0 release notes: https://github.com/rancher/rke/releases/tag/v1.4.0

Hello Everyone. I am scaling up a new control node on our rke cluster. Rancher is able to provision the vm and start docker. But it only starts the first container and then stops. The first container is the rancher agent "rancher/rancher-agent:v2.6.13 "run.sh --server htt…"" Any tips on what is failing? Or how to get this to move forward? We have 2 control nodes currently running. We deleted one of them to upgrade and now we cannot spin up another one. The logs of this container are:

INFO: Arguments: --server <https://rancher.nameofplatform.com> --token REDACTED -r -n m-ccnpb
INFO: Environment: CATTLE_ADDRESS=10.217.43.18 CATTLE_AGENT_CONNECT=true CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME=m-ccnpb CATTLE_SERVER=<https://rancher.nameofplatform.com> CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver 127.0.0.53 options edns0 trust-ad search <http://3xjoa1i2lshejpt0ninj35opia.bx.internal.cloudapp.net|3xjoa1i2lshejpt0ninj35opia.bx.internal.cloudapp.net>
WARN: Loopback address found in /etc/resolv.conf, please refer to the documentation how to configure your cluster to resolve DNS properly
INFO: <https://rancher.nameofplatform.com/ping> is accessible
INFO: <http://rancher.nameofplatform.com|rancher.nameofplatform.com> resolves to 10.217.42.5
time="2023-08-17T15:57:24Z" level=info msg="Listening on /tmp/log.sock"
time="2023-08-17T15:57:24Z" level=info msg="Rancher agent version v2.6.13 is starting"
time="2023-08-17T15:57:24Z" level=info msg="Option etcd=false"
time="2023-08-17T15:57:24Z" level=info msg="Option controlPlane=false"
time="2023-08-17T15:57:24Z" level=info msg="Option worker=false"
time="2023-08-17T15:57:24Z" level=info msg="Option requestedHostname=m-ccnpb"
time="2023-08-17T15:57:24Z" level=info msg="Option dockerInfo={FUN4:2SZJ:AXCV:IW6F:JUDO:MHCD:WIJT:QGAB:PAE4:2VWN:GZCI:SGDX 1 1 0 0 1 overlay2 [[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] [] {[local] [bridge host ipvlan macvlan null overlay] [] [awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} true true false false true true true true true true true true false 32 false 40 2023-08-17T15:57:24.266091701Z json-file systemd 2 0 5.15.0-1041-azure Ubuntu 22.04.3 LTS 22.04 linux x86_64 <https://index.docker.io/v1/> 0xc001296150 2 8324927488 [] /var/lib/docker    rke-infra-control-4 [provider=azure] false 20.10.24   map[io.containerd.runc.v2:{runc [] <nil>} io.containerd.runtime.v1.linux:{runc [] <nil>} runc:{runc [] <nil>}] runc {  inactive false  [] 0 0 <nil> []} false  docker-init {8165feabfdfe38c65b599c4993d227328c231fca 8165feabfdfe38c65b599c4993d227328c231fca} {v1.1.8-0-g82f18fe v1.1.8-0-g82f18fe} {de40ad0 de40ad0} [name=apparmor name=seccomp,profile=default name=cgroupns]  [] []}"
time="2023-08-17T15:57:24Z" level=info msg="Option customConfig=map[address:10.217.43.18 internalAddress: label:map[] roles:[] taints:[]]"
time="2023-08-17T15:57:24Z" level=info msg="Connecting to <wss://rancher.nameofplatform.com/v3/connect> with token starting with 945w6gmwldmfpsadasda42knk6r"
time="2023-08-17T15:57:24Z" level=info msg="Connecting to proxy" url="<wss://rancher.nameofplatform.com/v3/connect>"
time="2023-08-17T15:57:24Z" level=info msg="Requesting kubelet certificate regeneration"
time="2023-08-17T15:57:24Z" level=info msg="Starting plan monitor, checking every 15 seconds"
time="2023-08-17T15:57:39Z" level=info msg="Requesting kubelet certificate regeneration"
time="2023-08-17T15:57:39Z" level=info msg="Plan monitor checking 120 seconds"

Has anyone been able to get alternate CIDRs to work in any version of RKE1?

ssh_key_path: /home/...
cluster_name: kubernetes
#kubernetes_version: 'v1.21.14-rancher1-1'
kubernetes_version: 'v1.24.16-rancher1-1'
enable_cri_dockerd: true
authorization:
  mode: rbac
#network:
#  plugin: calico
ingress:
  provider: none

services:
  kube-api:
    service_cluster_ip_range: 172.16.128.0/17
  kube_controller:
    cluster_cidr: 172.16.0.0/17
    service_cluster_ip_range: 172.16.128.0/17
  kubelet:
    cluster_dns_server: 172.16.128.10

nodes:
  - address: 10.75.144.169
    user: rke
    role:
      - etcd
      - controlplane

Hi, needing advice: One of my RKE 1.3.11 nodes went away because a disks crash. I had to replace it and reprovision the OS with ansible. Not sure what the procedure is to have that node back to life, please confirm if I am right or not with the next steps: • Shall I remove the node from the cluster with a "kubectl delete node k8s-10.10.10.43" before proceeding to the next step? • run "rke up --config myconfig.yaml --update-only" I am not sure if with just update only will not break the production cluster.

fortunatelly it is a worker node. but I am wondering what would happen if a master node fails. Will the etcd and control plane blown up if I run "rke up --config myconfig.yaml" ??

Hello -- just wandering what is major changelog between v1.4 and new v1.5 RKE version?

Hello, dose anybody tried to use the feature "seccompDefault" with RKE1?

Hello everyone I opened up a github ticket with no response. I am not able to create a new control node. https://github.com/rancher/rke/issues/3336 Can someone please take a look?

Hello I am trying to deploy a new control node for RKE1 cluster. It creates the VM and then just sits and says waiting to register with kubernetes. Does anyone know if I need to manually approve this node as a control node? Or if my kube-controller-manager is pointing to a deleted node and it will not allow any new nodes in the cluster? Here is a bit more info: https://github.com/rancher/rke/issues/3336

Hi, the RKE1 “How Upgrades Work” docs mention that starting with v1.1.0, you can “manually upgrade nodes of a certain role“. I see no further reference to that, and while v1.1.0 mentions changes to the entire upgrade logic, I don’t see something regarding a “manual upgrade” procedure specifically. Does anyone know what that’s referring to?

wondering if anyone can point me in the right direction for modifying the RKE2 registries file (to add a private registry) when using Elemental. I can write a file to the same directory

/etc/rancher/rke2

but the

registries.yaml

file always ends up back to the default (oops wrong channel!)

@echoing-tomato-53055 has left the channel

Hello Community, Sorry for the silly question, would appreciate it if someone could clarify the confusion. I see the max version listed on the Support Matrix for RKE1 as 1.25, can someone assist in confirming if 1.25 is the last supported version we have for RKE1, and for further Kubernetes versions do we need to migrate to RKE2? Support Matrix link: https://www.suse.com/suse-rke1/support-matrix/all-supported-versions/rke1-v1-25/

@acoustic-addition-45641 has left the channel

Hello #rke

Any one having documents or info of how to use haproxy as load balancer for deployments in rke.

hello why i see my cluster cls-tst-001 only in mgmt clusters and not in clusters ? someone know because i want upgrade my cluster cls-tst-001 kubernetes at 1.22

Hi all when im tring to add new worker im getting error no live upstreams while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:6443, upstream: "kube_apiserver", bytes from/to client:0/0, bytes from/to upstream:0/0

I installed a new cluster using rancher but my rancher server does no longer work. Is it possible to get the kubeconfig from the controlplane so I can access the cluster?

Hi i am currently deploying downstream clusters in rancher 2.7.5 with RKE1 on Nutanix with the official Nutanix docker machine driver. Upon node installation and execution of the docker install script I get this error Message in Rancher: Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "10.105.1.90:2376": Get "https://10.105.1.90:2376/version": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Where can I check what certificate gets used for this endpoint?

Hello guys im trying to import an rke k8 cluster v1.24 as a generic cluster in rancher v2.7.3 and once i run the rancher agent install script on my rke cluster the agent is stuck with the below message:

Requesting kubelet certificate regeneration

i have tried rebuilding the rke cluster and re-installling the agent i still keep getting the same error. any ideas on what i could be doing wrong? note im creating rke cluster using the rke cli and not through rancher.

Hello, I've installed RKE1(1Master+1worker) on two RHEL servers, but DNS resolution taking very long time, 5-25 seconds via cluster cluster service. I can reach CoreDNS pod running on same server, but can't reach coredns pod from other server. FW is disabled. Any idea how to troubleshoot further ?

@prehistoric-gpu-28362 has left the channel

I’ve created a new node pool and added some nodes via the Rancher UI (v2.7.5) - I’m seeing that the kubelet that is started on these new nodes is using

--root-dir=/opt/rke/var/lib/kubelet

whereas all the previous nodes are using

--root-dir=/var/lib/kublet

this is causing the AWS aws-ebs-csi-driver to not be able to run on the new nodes - anyone have any thoughts on why/how this changed?

anyone aware of any documentation around restoring a kube-controller-manager? some sort of system contention happened and my kube-controller-manager was completely removed from docker (cant find logs around this), which means my deployments arent functioning, and I cant get a new cattle-cluster-agent to spawn, so I cant use the standard rancher UI path to restore the cluster from snapshot. I have an etcd backup as well (this is a rancher spawned cluster), we could totally restore from a snapshot, but I havent seen documentation that doesnt involve using the UI -- which requires a functioning cattle-cluster-agent ;'(