https://rancher.com/ logo
Join Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
rke2
  • f

    future-grass-75901

    04/29/2022, 2:26 PM
    Hello Fellow Rancher Users ... currently I am investigating EGress solutions and wanted to test Cilium ... as described on https://docs.cilium.io/en/latest/gettingstarted/k8s-install-helm/#install-cilium it should be possible via RKE and RKE2 (but apparently ist natively supported by RKE2 and recommended using the latter) ... but since RKE2 is viewed as an Technical Preview in Rancher 2.6.4 I wanted to use RKE since I have some expirience there ... but it showed some strange behaviour that the cilium health checks were not properly running and it was not very stabel with all that pod-starting-before-CNI-taint-stuff ... also first I setup a single node cluster just for testing which worked eventually after specing the VM to at least 4CPU/8GRam and restarting the pods and cilium daemonset ... anyhow after wanting to grow it to 3Nodes things got strange again so I wanted to try the native support from RKE2 ... first with a single node setup - But again the same issues coredns/cattle-cluster-agent pods crashlooping and cilium health check not properly working another round of rollout restart ds cilium && restarting unmanaged pods and some time did the trick ... for me this looked like the DNS pods started bevore the CNI therefore they had no network and the cattle-clsuter-agent could not resolve and connect to rancher ... today I updated the hypercube version from v1.21.10 > v1.21.12 and it seems to work ... but all this is quite strange and I would expect that Rancher would launch the nodes with this ominous taint cilium is recommending > https://docs.cilium.io/en/latest/gettingstarted/taints/ ... TLDR: is there some good tutorial on how to setup an k8s cluster via the rancher UI with the help of RKE2 :?
    s
    c
    • 3
    • 15
  • a

    acceptable-gigabyte-29847

    04/29/2022, 3:09 PM
    Hello hello, what version of RKE2 should I use to not get any “-buildxxxxx” images? I’m working in an air gapped environment and I’m trying to figure out which images I need. However I see a lot of “-buildxxxx” images…
    c
    • 2
    • 4
  • b

    bland-jackal-22983

    05/01/2022, 12:52 AM
    another question, regarding kubelet extra blinds: previously in 
    rke
    , we need to use 
    services_kubelet->extra-binds
    to add a mapping in 
    cluster.yaml
    like: 
    /opt/rke/var/lib/kubelet/plugins:/var/lib/kubelet/plugins
    specifically i am trying to do this just wondering how can we add the extra binds if it's 
    rke2
    , do i need to? i can't find corresponding server/agent config flag for this in the rke2 doc
    c
    • 2
    • 3
  • m

    mysterious-parrot-80714

    05/01/2022, 6:16 PM
    and pushing the default policies don't help 😕
    g
    • 2
    • 1
  • f

    future-truck-59205

    05/02/2022, 8:19 AM
    Hi everyone, I've been trying to configure Azure credentials in Rancher but every time I try it gives me "Authentication test failed, please check your credentials". Has anyone else also faced the same problem and how did you get rid of it? I've created the correct steps as given on docs ie: https://rancher.com/docs/rancher/v2.5/en/cluster-provisioning/rke-clusters/cloud-providers/azure/
    c
    • 2
    • 2
  • a

    acceptable-gigabyte-29847

    05/02/2022, 10:17 AM
    Any ideas how I can provide my own CA to the rancher-agent for communication to the rancher server? I can’t seem to find anything in the docs.
    ✅ 1
    • 1
    • 2
  • w

    wonderful-helicopter-16401

    05/03/2022, 4:23 AM
    Hi, I am doing my first install of Rancher on RKE2 (2.6) and am having some issues with accessing 
    /var/lib/rancher/rke2/*server/manifests*
    to configure nginx as a daemonset as outlined in these docs. It appears the permissions for the 
    server
     directory are not set properly on any of the nodes in this cluster. I'm seeing the steps were tested in 2.5.6. So, is this a bug or intended? Also, what permissions should I set on that directory to complete the final step in those docs?
    c
    r
    • 3
    • 15
  • n

    numerous-country-20400

    05/03/2022, 11:21 AM
    How do i retrieve the kube-proxy network pool under rke2? Asking to resolve an issue with calico at https://github.com/projectcalico/calico/issues/5952#issuecomment-1115971887
    c
    • 2
    • 4
  • f

    faint-airport-83518

    05/04/2022, 2:39 PM
    is there any way to use the automated upgrade controller in an airgap?
    g
    • 2
    • 127
  • m

    magnificent-caravan-81252

    05/04/2022, 3:47 PM
    Hey friends! Can someone help clear up some confusion for me about the installation docs? As part of the process of installing a Rancher cluster using RKE2, it says that you need to setup “two” listeners on port 9345 and port 6443. However, the link shown for setting up the LB in a previous step doesn’t mention anything other than port 80 and 443. Am I to understand that I should add 9345 and 6443 as listeners onto the LB? I’m a little nervous about the security implications.
    c
    r
    +2
    • 5
    • 12
  • n

    narrow-noon-75604

    05/05/2022, 11:06 AM
    Hi, I am trying to add the agents (worker) nodes to the rke2 server, but the "rke2-agent" service is stuck in "activating" state with an error msg "failed to get CA certs: ". I am unable to find any recommendations for the issue. So please help me in fixing this issue.
    b
    • 2
    • 16
  • c

    creamy-scooter-43304

    05/05/2022, 1:37 PM
    Could You please share with some example for rke2 config file ?
    b
    • 2
    • 21
  • a

    acceptable-gigabyte-29847

    05/05/2022, 3:21 PM
    Hello, is it possible to deploy a Rancher Server on top of a cluster (RKE2) deployed by another Rancher Server?
    b
    g
    w
    • 4
    • 11
  • c

    curved-caravan-26314

    05/07/2022, 10:19 PM
    Hello all, is it difficult to turn a rke2 from single server to HA? I can set 3 replicas on the first server and then add the other to control planes later on?
    c
    • 2
    • 3
  • r

    ripe-queen-73614

    05/08/2022, 9:23 AM
    Please can someone help me? Thank you
    b
    • 2
    • 2
  • c

    curved-caravan-26314

    05/08/2022, 6:04 PM
    Does anyone know if rke2 would be able to leverage an external etcd mysql database like K3S?
    c
    • 2
    • 3
  • h

    hallowed-breakfast-56871

    05/08/2022, 11:12 PM
    Hey folks - Anyone got filebeat working with collecting pod logs with RKE2? I see logs are placed / setup differently on RKE2 so this means filebeat gets confused with this logs belongs to which pod. https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html
    c
    • 2
    • 7
  • c

    curved-caravan-26314

    05/08/2022, 11:25 PM
    Im on a new stable install of rancher and I am getting this error and i have no backup to get a new secret. what do i do? Ive been through the troubleshooting page and everything else checks out. 
    Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    45s                default-scheduler  Successfully assigned cattle-system/rancher-webhook-6958cfcddf-jzqd7 to k42
  Warning  FailedMount  45s (x2 over 45s)  kubelet            MountVolume.SetUp failed for volume "tls" : secret "rancher-webhook-tls" not found
  Normal   Pulled       41s                kubelet            Container image "rancher/rancher-webhook:v0.2.5" already present on machine
  Normal   Created      40s                kubelet            Created container rancher-webhook
  Normal   Started      39s                kubelet            Started container rancher-webhook
    c
    • 2
    • 8
  • s

    sparse-continent-53433

    05/09/2022, 2:00 PM
    Are there any directions for fixing a host that has expired certificates? I'm receiving this error on a few of my nodes: 
    level=error msg="CA cert validation failed: Get \"<https://127.0.0.1:6444/cacerts>\": x509: certificate has expired or is not yet valid
    c
    • 2
    • 8
  • h

    hundreds-airport-66196

    05/09/2022, 9:03 PM
    Our rke2 is up and running in prod. Now, it has having conflict with network subnet 10.42.0.0/16 as this subnet is also use by the cluster load. Is there a documentation on how to change the cidr settings? thanks
    c
    • 2
    • 14
  • b

    busy-orange-63415

    05/09/2022, 11:53 PM
    Hi, can someone help me? https://rancher-users.slack.com/archives/C3ASABBD1/p1652140228814169
    b
    • 2
    • 1
  • s

    silly-jordan-81965

    05/10/2022, 11:31 AM
    What do i need to change to get rke2 to give me logs in json format instead of klog?
    b
    • 2
    • 2
  • f

    faint-ram-5049

    05/11/2022, 7:05 PM
    Does anyone know if there is documentation showing the maximum number of worker nodes RKE2 can maintain?
    c
    r
    • 3
    • 5
  • v

    victorious-analyst-3332

    05/12/2022, 12:59 AM
    Would anyone be able to tell me if it is possible to install arbitrary CRDs as a part of the RKE2 install process? We’re currently installing calico via Helm chart to stay more recent on RKE installs, but it appears that we can do most of that via RKE2 Helm Integration. In our case, we need to apply the calico CRDs and create a number of custom resources for things like BGP peering and felixConfig before the operator deploy (options not present in the operator), so I was hoping there might be a method for also applying the custom config. I did see the 
    /var/lib/rancher/rke2/server/manifests
    approach, but don’t fully understand the ramifications of the 
    AddOn custom resources
    detail mentioned there. Thanks a lot for any help you can give.
    c
    c
    • 3
    • 23
  • s

    silly-jordan-81965

    05/13/2022, 7:45 AM
    Im deploying a rancher2_cluster_v2 cluster and want to add cloud provider config. But receives the following error when doing that: on main.tf line 109, in resource “rancher2_cluster_v2” “k8s-sandbox-sc-mr”: │ 109: cloud_provider { │ │ Blocks of type “cloud_provider” are not expected here. My terraform code looks like this: 
    rke_config {
    cloud_provider {
      name = "openstack"
      openstack_cloud_provider {
        global {
          auth_url = var.openstack_auth_url
          username = var.openstack_username
          password = var.openstack_password
          tenant_id = var.openstack_tenant_id
        }
      }
    }
    machine_global_config = <<EOF
    Anyone that can point me in the right direction?
    l
    • 2
    • 3
  • r

    rapid-helmet-86074

    05/13/2022, 1:32 PM
    I noticed today that Rancher 2.6.5 was released (https://forums.rancher.com/t/rancher-release-v2-6-5/37891/3) and in it RKE2 deployment goes from tech preview to general availability, but that's only with Kubernetes v1.22+. Can any Rancher folks let me know if there're plans to extend that to earlier Kubernetes versions? We've got a reason that we were hoping to use v1.20 at the moment, so I'm wondering if my best answer is wait or push a completely separate team to upgrading Kubernetes versions.
    c
    • 2
    • 4
  • r

    refined-magician-25478

    05/13/2022, 3:48 PM
    Does anyone know if you can use the Ranchers helm crd with a local helm chart, not in a chart repository?
    c
    • 2
    • 14
  • h

    hundreds-airport-66196

    05/13/2022, 10:08 PM
    How to upgrade rke2 on an airgap installation?
    g
    • 2
    • 5
  • b

    broad-farmer-70498

    05/16/2022, 8:07 PM
    anyone around been able to get host-process (windows) containers working with rke2? I'm not having much luck..
    m
    j
    • 3
    • 20
  • b

    bland-jackal-22983

    05/17/2022, 8:55 AM
    hi, i couldn't figure out why my 
    rke2-server
    installation would fail when i set the 
    node-label
    values? in the 
    /etc/rancher/rke2/config.yaml
    i have: 
    node-label:
- <http://node-role.kubernetes.io/master|node-role.kubernetes.io/master>
- <http://openebs.io/engine=mayastor|openebs.io/engine=mayastor>
    now if i install rke2 with everything the same as before but with this new 
    node-label
    , i would have this error: 
    May 17 08:48:26 698F191 rke2[121465]: time="2022-05-17T08:48:26Z" level=error msg="Kubelet exited: exit status 1"
May 17 08:48:28 698F191 rke2[121465]: {"level":"warn","ts":"2022-05-17T08:48:28.923Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000955340/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
May 17 08:48:28 698F191 rke2[121465]: time="2022-05-17T08:48:28Z" level=info msg="Failed to test data store connection: context deadline exceeded"
May 17 08:48:30 698F191 rke2[121465]: {"level":"warn","ts":"2022-05-17T08:48:30.005Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000955340/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
May 17 08:48:30 698F191 rke2[121465]: time="2022-05-17T08:48:30Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
May 17 08:48:30 698F191 rke2[121465]: time="2022-05-17T08:48:30Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    if i don't have this 
    node-label
    section, the installation goes through. os: 
    ubuntu 20..04
    rke2 version: 
    v1.22.9
    c
    • 2
    • 6
Title
b

bland-jackal-22983

05/17/2022, 8:55 AM
hi, i couldn't figure out why my 
rke2-server
installation would fail when i set the 
node-label
values? in the 
/etc/rancher/rke2/config.yaml
i have: 
node-label:
- <http://node-role.kubernetes.io/master|node-role.kubernetes.io/master>
- <http://openebs.io/engine=mayastor|openebs.io/engine=mayastor>
now if i install rke2 with everything the same as before but with this new 
node-label
, i would have this error: 
May 17 08:48:26 698F191 rke2[121465]: time="2022-05-17T08:48:26Z" level=error msg="Kubelet exited: exit status 1"
May 17 08:48:28 698F191 rke2[121465]: {"level":"warn","ts":"2022-05-17T08:48:28.923Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000955340/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
May 17 08:48:28 698F191 rke2[121465]: time="2022-05-17T08:48:28Z" level=info msg="Failed to test data store connection: context deadline exceeded"
May 17 08:48:30 698F191 rke2[121465]: {"level":"warn","ts":"2022-05-17T08:48:30.005Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000955340/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
May 17 08:48:30 698F191 rke2[121465]: time="2022-05-17T08:48:30Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
May 17 08:48:30 698F191 rke2[121465]: time="2022-05-17T08:48:30Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
if i don't have this 
node-label
section, the installation goes through. os: 
ubuntu 20..04
rke2 version: 
v1.22.9
c

creamy-pencil-82913

05/17/2022, 2:31 PM
Can you open a GH issue, and attach the rke2-server logs and everything from /var/log/pods? The messages you quoted are all normal to see while things are initially starting up. Also, the master role label is automatically set by rke2 on servers, why are you specifying it manually?
b

bland-jackal-22983

05/17/2022, 3:28 PM
thanks! will create a issue for this. but for the rke2-sever logs, there is a lot 
flag is deprecated
messages and keep generating: 
May 17 15:16:07 698F191 rke2[214445]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --sync-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
May 17 15:16:07 698F191 rke2[214445]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
May 17 15:16:07 698F191 rke2[214445]: Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --eviction-minimum-reclaim has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
May 17 15:16:07 698F191 rke2[214445]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
i understand what it means - put them in a kubelet config file and use 
--config
pointing to the file, instead of directly specifying them to 
kubelet-args
. however, i never explicitly specify them. is this expected? lastly, after i remove the 
<http://node-role.kubernetes.io/master|node-role.kubernetes.io/master>
label, the rke2-server installation succeeded.
c

creamy-pencil-82913

05/17/2022, 3:46 PM
Yes the warnings about flags being deprecated is to be expected.
❤️ 1
I can't recall for sure, but I believe that the node-role labels are blocklisted by upstream Kubernetes to prevent nodes from setting those on themselves. Why were you trying to do that?
As I described above, it's not necessary.
👍 1
b

bland-jackal-22983

05/17/2022, 3:50 PM
thanks for the info. i didn't know it's set automatically, it's not necessary
View count: 11