Hello Guys, I'm trying to re-create my org env in my rancher desktop local i.e using calico-policy-only.yaml alongside flannel. But for some reason calico is failing to create/maintain the network interfaces (cali* prefixed) in the worker node, is there any setting from rancher side we need to handle ? Some restriction is blocking calico to create so ? Due to this , calico-node daemonset is not starting since felix is failing to be ready This is the issue I had raised in calico github - https://github.com/projectcalico/calico/issues/7782 Do let me know if you have any input , if rancher env has to enable any permission or so.

Hi, Iv installed a new rancher ( v1.25.10+k3s1 ) with HELM on ubuntu 22.04 and I'm trying to create a new cluster on ubuntu 22.04 ( Custom ) im getting error on the node service Jun 19 135749 k8s-lab rancher-system-agent[1258]: time="2023-06-19T135749Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/serv> Jun 19 135749 k8s-lab rancher-system-agent[1258]: time="2023-06-19T135749Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Hello y'all. I have upgraded my kubernetes version with an RKE1 template to kubernetes 1.24 and now none of my Persistent Volume Claims will mount with my pods. We are using azure for our PVCs and I made sure the template has the correct azure service principal to be able to connect. Does anyone have any experience with troubleshooting this? The error looks like this:

Failed Mount 
Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[ready-files kube-api-access-8jhkt data-0 app-name-tmp cluster-ca broker-certs client-ca-cert kafka-metrics-and-logging]: timed out waiting for the condition

If I need to migrate my Rancher installation that runs on 3 RKE nodes, all with control plane, etcd and worker roles, what are my options for this? Can I just spin up new nodes and add them to my nodes list on my rancher-cluster.yml? Are there any special steps to follow for migrating etcd?

Hi. I'm using RKE1 to init my K8s cluster. I know that RKE2 is the next step, but migrating to RKE2 in on-premise infra is pretty tough. So I have a few questions: • RKE1 stable and long-term support? • Should we upgrade to RKE2 for some kind of reasons? Is there any good solution for on-premise infra? • Is using RKE1 can be consider as a tech debt and we have to resolve it?

I'm scratching my head... is there anything in RKE1 that's rotating container logs? If not (by default) is there a way to configure something to rotate them?

Hello everyone, I am looking for some help. Our environment is RKE1 on azure VMs. We are using kubernetes 1.24 and rancher 2.6.13. We are having a problem attaching or mounting volumes to azure disk. We are getting a timeout "timed out waiting for the condition" and the other error message we are getting is :

AttachVolume.Attach failed for volume "pvc-examplename" : timed out waiting for external-attacher of <http://file.csi.azure.com|file.csi.azure.com> CSI driver to attach volume resource-group-name#rkeinfra43b5b092#rke-infra-dynamic-pvc-examplename#pvc-examplename#namespace-example

We reset the service principal password but still doesn't seem to be connecting. We are opened a case with Azure but not getting much help either, we are not using AKS. Does anyone have any ideas about this issue or have dealt with anything like this in the past? Thanks in advance.

I also created an incident on the Rancher github for the above issue. https://github.com/rancher/rancher/issues/42072

I have upgraded my custom rke cluster from version 1.23 to 1.24 by upgrading Rancher to version 2.74 (the latest docker image available at that time) and the upgraded k8s cluster through its menu options. Since k8s 1.24 removed docker runtime for containers, does it mean that Rancher took care of it and provided different container runtime? How can I check that? I am running the cluster on prem on a bunch of RHEL8 VMs

I’m creating RKE cluster on Azure RHEL7 VM and have no issues. When I used Azure RHEL7 VM that IT provided to me it is failing at deploy job, when I exec to the pod for deploy and run kubectl it says connection to the server refused. What can be an issue?

Creating ELB on openstack. It’s in this state forever, any guides?

@faint-shampoo-17603 has left the channel

@late-lunch-73194 has left the channel

Hi All, I need some advice I have installed a new rancher ( the latest version ) and created a new cluster (Custom, based on ubuntu 22.04 ) when I'm trying to register a new machine ( based on ubuntu 22.04, v1.26.6+rke2r1 ) it get stuck on the provisioning log i can see : [INFO ] waiting for at least one control plane, etcd, and worker node to be registered [INFO ] configuring bootstrap node(s) custom-ff5bc7cc5078: waiting for agent to check in and apply initial plan [INFO ] configuring bootstrap node(s) custom-ff5bc7cc5078: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet [INFO ] configuring bootstrap node(s) custom-ff5bc7cc5078: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler [INFO ] configuring bootstrap node(s) custom-ff5bc7cc5078: waiting for probes: kube-apiserver, kube-controller-manager, kube-scheduler [INFO ] configuring bootstrap node(s) custom-ff5bc7cc5078: waiting for cluster agent to connect and in the machine agent log i see : Aug 01 122336 lab-k8s rancher-system-agent[1162]: time="2023-08-01T122336Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler" Aug 01 122336 lab-k8s rancher-system-agent[1162]: time="2023-08-01T122336Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory" Aug 01 122336 lab-k8s rancher-system-agent[1162]: time="2023-08-01T122336Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" is this a bug ? am I'm doing something wrong ? i can see issue opened here https://github.com/rancher/rancher/issues/41650 10x

also i can see in /var/lib/rancher/rke2/agent/logs/kubelet.log failed (add): failed to allocate for range 0: no IP addresses available in range set

Forgive me for this possibly dumb question... I heard someone saying that latest version of RKE1 no longer support Windows worker node? Is that correct?

Hello Everyone. I am using RKE1. I updated my node template to Ubuntu 22 LTS image for the vm (azure). And then I am trying to deploy a new control node and I am having issues. It just tries to create a node and then fails after 10 minutes and tries again. Does anyone have any tips on how to debug this?

Hope everyone had a great weekend. I was wondering if anyone had any thoughts on my post above ^^ Thanks in advance. 🙂

I have rancher v2.7.5 running from docker container (official) and imported Digital Ocean managed kubernetes v1.24.16. Now that DO is going to force update the Kubernetes and the upgrade check comes up with an error. How do I fix this? Thank you.

Validating webhook is configured in such a way that it may be problematic during upgrades.

Resources

validating webhook configuration: <http://rancher.cattle.io|rancher.cattle.io>

Does deleting the Imported Kubernetes within Rancher UI, removes all the Rancher installed namespace in the Kubernetes Cluster?

Hello Everyone. I am trying to restart a Control Plane node. We currently have 2 control nodes running and one deleted. There seems to be an etcd issue and it wont start up whenever we spin up a new node. Docker runs "rancher/rancher-agent:v2.6.13 "run.sh --server htt…"" this container but it gets stuck "Requesting kubelet certificate regeneration". Our cluster is stuck in Updating state with the error: "waiting for control-1 to finish provisioning." We are able to see that our

kube-controller-manager

is set to that node and is currently deleted. Is there anyway to change the kube-controller-manager or get etcd issue to fix so we can redeploy this node? Basically our cluster is stuck waiting to provision a node that doesn't exist, and when we try to create it, it fails.

$ kubectl describe endpoints kube-controller-manager  -n kube-system
Name:         kube-controller-manager
Namespace:    kube-system
Labels:       <none>
Annotations:  <http://control-plane.alpha.kubernetes.io/leader|control-plane.alpha.kubernetes.io/leader>:
                {"holderIdentity":"control-1","leaseDurationSeconds":15,"acquireTime":"2022-01-11T16:00:56Z...
Subsets:
Events:  <none>

Does anyone have any clue on how to debug my above post? Or how to get our new control node back up? Would really appreciate any help.

Is there any way to configure Garbage Collection for images and containers via kubelet settings in RKE1?

Couldn’t find it in the docs

Hi! How come RKE1 has 1.3.x and 1.4.x releases concurrently? Is there a reason not to use 1.4.x (coming from 1.3.x)? I don’t see anything in the 1.4.0 release notes: https://github.com/rancher/rke/releases/tag/v1.4.0

Hello Everyone. I am scaling up a new control node on our rke cluster. Rancher is able to provision the vm and start docker. But it only starts the first container and then stops. The first container is the rancher agent "rancher/rancher-agent:v2.6.13 "run.sh --server htt…"" Any tips on what is failing? Or how to get this to move forward? We have 2 control nodes currently running. We deleted one of them to upgrade and now we cannot spin up another one. The logs of this container are:

INFO: Arguments: --server <https://rancher.nameofplatform.com> --token REDACTED -r -n m-ccnpb
INFO: Environment: CATTLE_ADDRESS=10.217.43.18 CATTLE_AGENT_CONNECT=true CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME=m-ccnpb CATTLE_SERVER=<https://rancher.nameofplatform.com> CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver 127.0.0.53 options edns0 trust-ad search <http://3xjoa1i2lshejpt0ninj35opia.bx.internal.cloudapp.net|3xjoa1i2lshejpt0ninj35opia.bx.internal.cloudapp.net>
WARN: Loopback address found in /etc/resolv.conf, please refer to the documentation how to configure your cluster to resolve DNS properly
INFO: <https://rancher.nameofplatform.com/ping> is accessible
INFO: <http://rancher.nameofplatform.com|rancher.nameofplatform.com> resolves to 10.217.42.5
time="2023-08-17T15:57:24Z" level=info msg="Listening on /tmp/log.sock"
time="2023-08-17T15:57:24Z" level=info msg="Rancher agent version v2.6.13 is starting"
time="2023-08-17T15:57:24Z" level=info msg="Option etcd=false"
time="2023-08-17T15:57:24Z" level=info msg="Option controlPlane=false"
time="2023-08-17T15:57:24Z" level=info msg="Option worker=false"
time="2023-08-17T15:57:24Z" level=info msg="Option requestedHostname=m-ccnpb"
time="2023-08-17T15:57:24Z" level=info msg="Option dockerInfo={FUN4:2SZJ:AXCV:IW6F:JUDO:MHCD:WIJT:QGAB:PAE4:2VWN:GZCI:SGDX 1 1 0 0 1 overlay2 [[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff false] [userxattr false]] [] {[local] [bridge host ipvlan macvlan null overlay] [] [awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} true true false false true true true true true true true true false 32 false 40 2023-08-17T15:57:24.266091701Z json-file systemd 2 0 5.15.0-1041-azure Ubuntu 22.04.3 LTS 22.04 linux x86_64 <https://index.docker.io/v1/> 0xc001296150 2 8324927488 [] /var/lib/docker    rke-infra-control-4 [provider=azure] false 20.10.24   map[io.containerd.runc.v2:{runc [] <nil>} io.containerd.runtime.v1.linux:{runc [] <nil>} runc:{runc [] <nil>}] runc {  inactive false  [] 0 0 <nil> []} false  docker-init {8165feabfdfe38c65b599c4993d227328c231fca 8165feabfdfe38c65b599c4993d227328c231fca} {v1.1.8-0-g82f18fe v1.1.8-0-g82f18fe} {de40ad0 de40ad0} [name=apparmor name=seccomp,profile=default name=cgroupns]  [] []}"
time="2023-08-17T15:57:24Z" level=info msg="Option customConfig=map[address:10.217.43.18 internalAddress: label:map[] roles:[] taints:[]]"
time="2023-08-17T15:57:24Z" level=info msg="Connecting to <wss://rancher.nameofplatform.com/v3/connect> with token starting with 945w6gmwldmfpsadasda42knk6r"
time="2023-08-17T15:57:24Z" level=info msg="Connecting to proxy" url="<wss://rancher.nameofplatform.com/v3/connect>"
time="2023-08-17T15:57:24Z" level=info msg="Requesting kubelet certificate regeneration"
time="2023-08-17T15:57:24Z" level=info msg="Starting plan monitor, checking every 15 seconds"
time="2023-08-17T15:57:39Z" level=info msg="Requesting kubelet certificate regeneration"
time="2023-08-17T15:57:39Z" level=info msg="Plan monitor checking 120 seconds"

Has anyone been able to get alternate CIDRs to work in any version of RKE1?

ssh_key_path: /home/...
cluster_name: kubernetes
#kubernetes_version: 'v1.21.14-rancher1-1'
kubernetes_version: 'v1.24.16-rancher1-1'
enable_cri_dockerd: true
authorization:
  mode: rbac
#network:
#  plugin: calico
ingress:
  provider: none

services:
  kube-api:
    service_cluster_ip_range: 172.16.128.0/17
  kube_controller:
    cluster_cidr: 172.16.0.0/17
    service_cluster_ip_range: 172.16.128.0/17
  kubelet:
    cluster_dns_server: 172.16.128.10

nodes:
  - address: 10.75.144.169
    user: rke
    role:
      - etcd
      - controlplane

Hi, needing advice: One of my RKE 1.3.11 nodes went away because a disks crash. I had to replace it and reprovision the OS with ansible. Not sure what the procedure is to have that node back to life, please confirm if I am right or not with the next steps: • Shall I remove the node from the cluster with a "kubectl delete node k8s-10.10.10.43" before proceeding to the next step? • run "rke up --config myconfig.yaml --update-only" I am not sure if with just update only will not break the production cluster.

fortunatelly it is a worker node. but I am wondering what would happen if a master node fails. Will the etcd and control plane blown up if I run "rke up --config myconfig.yaml" ??

Hello -- just wandering what is major changelog between v1.4 and new v1.5 RKE version?