https://rancher.com/ logo
Docs
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
rke2
  • b

    big-jordan-45387

    03/07/2023, 12:57 PM
    hi, I have a small cluster of 4 nodes (x3 control plane & etcd & worker and x1 etcd and worker) and all of them shut down. I booted them and the 3 control planes are ok meaning I can kubectl get pods, etc. the 4rth one does not come up, rancher-system-agent is up and running but the rke2-agent is not up. Logs says:
    Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation.
    Waiting to retrieve agent configuration; server is not ready: failed to retrieve configuration from server: <https://127.0.0.1:6444/v1-rke2/config>: 401 Unauthorized
  • b

    big-jordan-45387

    03/07/2023, 12:57 PM
    I also have a rke2-server service, I am unsure whether I should try to bring that up too
    r
    h
    • 3
    • 5
  • m

    magnificent-vr-88571

    03/07/2023, 3:44 PM
    hi, I have RKE2 cluster deployed in a private network, internet connectivity can be reached through proxy server in aws. Followed https://docs.rke2.io/advanced#configuring-an-http-proxy and set proxy setting. With rke2-coredns using rancher/hardened-coredns:v1.9.3-build20220613 nslookup fails
    > kubectl run -it --rm test-pod --image=busybox:1.28 --restart=Never -- /bin/sh
    / # nslookup  kubernetes.default.svc.cluster.local
    Server:    10.43.0.10
    Address 1: 10.43.0.10 rke2-coredns-rke2-coredns.kube-system.svc.cluster.local
    
    Name:      kubernetes.default.svc.cluster.local
    Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local
    / # nslookup  kubernetes
    Server:    10.43.0.10
    Address 1: 10.43.0.10 rke2-coredns-rke2-coredns.kube-system.svc.cluster.local
    
    Name:      kubernetes
    Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local
    / # nslookup  <http://google.com|google.com>
    Server:    10.43.0.10
    Address 1: 10.43.0.10 rke2-coredns-rke2-coredns.kube-system.svc.cluster.local
    
    nslookup: can't resolve '<http://google.com|google.com>'
    a
    • 2
    • 8
  • h

    hundreds-evening-84071

    03/07/2023, 8:27 PM
    has anyone tried to deploy RKE2 cluster on RHEL 9? If so, any issues?
    c
    a
    a
    • 4
    • 7
  • j

    jolly-eye-77963

    03/08/2023, 9:35 AM
    Is there an example of an RKE2 config file (usable by the Rancher UI tool) where the scheduler has been changed to another kube-scheduler image - or something similar?
  • a

    abundant-noon-17295

    03/09/2023, 6:57 AM
    any insight on this https://cluster-api.sigs.k8s.io/ from Rancher’s point of view?
  • j

    jolly-eye-77963

    03/09/2023, 9:07 AM
    Is there a skeleton full RKE2 config.yaml or guide to customising images like the scheduler?
    c
    b
    • 3
    • 8
  • h

    hundreds-airport-66196

    03/09/2023, 5:20 PM
    Hi All, Anybody is using Calico as the rke2 CNI instead of Canal? How does it compare to the default CNI?
    h
    c
    • 3
    • 6
  • l

    loud-eve-73457

    03/10/2023, 3:39 AM
    is there a way to limit the outgoing network bandwidth of a pod to some IP address range. great thanks
    c
    • 2
    • 2
  • j

    jolly-eye-77963

    03/10/2023, 11:25 AM
    is there a way to set the default tolerations for all pods run in a namespace in RKE2?
  • s

    shy-zebra-53074

    03/10/2023, 3:55 PM
    Hey all! Looking to apply some custom configurations to
    rke2-coredns
    following this guidance here: https://docs.rke2.io/networking#coredns I see a sample helm chart like the following:
    apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
    kind: HelmChartConfig
    metadata:
      name: rke2-coredns
      namespace: kube-system
    spec:
      valuesContent: |-
        nodelocal:
          enabled: true
    Where can I find more information regarding all of the possible chart values for RKE2 CoreDNS? Thank you!
    g
    • 2
    • 3
  • s

    shy-zebra-53074

    03/11/2023, 5:55 AM
    Hello! So I am trying to implement the
    extraSecrets
    config parameter: https://github.com/rancher/rke2-charts/blob/e29071e486dc0b987ed665d9ebe16cfdb681247c/charts/rke2-coredns/rke2-coredns/1.19.401/values.yaml#L231 My CoreDNS Helm looks like this:
    apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
              kind: HelmChartConfig
              metadata:
                name: rke2-coredns
                namespace: kube-system
              spec:
                valuesContent: |-
                  servers:
                  - zones:
                    - zone: .
                    port: 53
                    plugins:
                    - name: errors
                    - name: health
                      configBlock: |-
                        lameduck 5s
                    - name: ready
                    - name: kubernetes
                      parameters: cluster.local in-addr.arpa ip6.arpa
                      configBlock: |-
                        pods insecure
                        fallthrough in-addr.arpa ip6.arpa
                        ttl 30
                    - name: prometheus
                      parameters: 0.0.0.0:9153
                    - name: forward
                      parameters: . /etc/resolv.conf
                    - name: cache
                      parameters: 30
                    - name: loop
                    - name: reload
                    - name: loadbalance
              extraSecrets:
                - name: certs
                  mountPath: /etc/ssl/private/certs
    However I’m seeing this error in the logs when starting up:
    unknown field "extraSecrets"
    Mar 11 00:36:53 ip-192-168-0-10.us-gov-east-1.compute.internal rke2[1981]: I0311 00:36:53.877190    1981 event.go:294] "Event occurred" object="kube-system/rke2-coredns-config" fieldPath="" kind="Addon" apiVersion="<http://k3s.cattle.io/v1|k3s.cattle.io/v1>" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml\""
    Mar 11 00:36:53 ip-192-168-0-10.us-gov-east-1.compute.internal rke2[1981]: W0311 00:36:53.883804    1981 warnings.go:70] unknown field "extraSecrets"
    The secret has been created in the
    kube-system
    namespace:
    $ kubectl get secrets -n kube-system | grep certs
    certs                                                              Opaque               2      17m
  • s

    shy-zebra-53074

    03/11/2023, 6:02 AM
    Neeeevermind, my tab spacing is off 😅
  • s

    sparse-artist-18151

    03/13/2023, 4:52 AM
    Deploying RKE2 with Metallb Issue - IP gets assigned to the loadbalancer but no ARP entries are showing up under the interfaces, ARP ping and curl fails to the deployed loadbalancer IP + port IP Pool and L2Advertisement config
    apiVersion: <http://metallb.io/v1beta1|metallb.io/v1beta1>
    kind: IPAddressPool
    metadata:
      name: core-net-192.168.92.140-159
      namespace: metallb-system
    spec:
      addresses:
      - 192.168.94.140-192.168.94.159
    ---
    apiVersion: <http://metallb.io/v1beta1|metallb.io/v1beta1>
    kind: L2Advertisement
    metadata:
      name: metallb-pool
      namespace: metallb-system
    spec:
      ipAddressPools:
      - core-net-192.168.99.140-159
    How can we enable kubeproxy ipvs on the management cluster? (at the moment i only have one cluster with workernodes added to the management cluster)
    kubeproxy:
          extra_args:
            ipvs-scheduler: lc
            proxy-mode: ipvs
    do i need to deploy a separate cluster with workernodes for this? Thanks a lot for your input, if im on the wrong channel for these questions please let me know, i apologize in advance
    • 1
    • 1
  • s

    swift-cricket-47233

    03/13/2023, 6:56 PM
    Feels like a stupid question - but how do you upgrade Rancher on an RKE2 cluster? I followed the instructions to upgrade RKE2 (https://docs.rke2.io/upgrade/manual_upgrade) and that went fine, but how do I update Rancher...?
    c
    s
    s
    • 4
    • 18
  • s

    sparse-artist-18151

    03/13/2023, 9:47 PM
    how to add the following to the cluster.yaml
    kube-proxy-arg:
      - proxy-mode=ipvs
      - ipvs-strict-arp=true
    I added the following based on - link but metallb is still not ARPing with the Loadbalancer IP
    kubeproxy:
          extra_args:
            ipvs-scheduler: lc
            proxy-mode: ipvs
    c
    • 2
    • 5
  • s

    stale-painting-80203

    03/14/2023, 12:38 AM
    Rancher v2.7.0 with downstream RKE2 cluster. Not able to import an orphaned cluster into a new instance of Rancher. Import Existing -> Import any Kubernetes cluster. After issuing the import command on the cluster several pods go into CrashLoop and do not recover:
    /var/lib/rancher/rke2/bin/kubectl         --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f <https://rancher75182.senode.dev/v3/import/xhctfcnbbt56xvxh6jptq7lzvpw9svd2drkbj5pvm466t5r7zlplqv_c-m-zqcvzlgn.yaml>
    <http://clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver|clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver> unchanged
    <http://clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master|clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master> unchanged
    namespace/cattle-system unchanged
    serviceaccount/cattle unchanged
    <http://clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding|clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding> unchanged
    secret/cattle-credentials-ad9a794 created
    <http://clusterrole.rbac.authorization.k8s.io/cattle-admin|clusterrole.rbac.authorization.k8s.io/cattle-admin> unchanged
    Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: <http://beta.kubernetes.io/os|beta.kubernetes.io/os> is deprecated since v1.14; use "<http://kubernetes.io/os|kubernetes.io/os>" instead
    deployment.apps/cattle-cluster-agent configured
    service/cattle-cluster-agent unchanged
    
    NAMESPACE             NAME                                                    READY   STATUS             RESTARTS      AGE
    calico-system         calico-kube-controllers-f75c97ff6-fvb66                 1/1     Running            0             19m
    calico-system         calico-node-6vxmh                                       1/1     Running            0             19m
    calico-system         calico-node-d9t8n                                       0/1     Running            0             17m
    calico-system         calico-node-khhpr                                       1/1     Running            0             19m
    calico-system         calico-node-nmcds                                       0/1     Running            0             17m
    calico-system         calico-typha-d65458ffc-97pn9                            1/1     Running            0             17m
    calico-system         calico-typha-d65458ffc-p9cj2                            1/1     Running            0             19m
    cattle-fleet-system   fleet-agent-6c857b85b5-zff2l                            1/1     Running            0             17m
    cattle-system         cattle-cluster-agent-6f588568-dj7ql                     0/1     CrashLoopBackOff   4 (49s ago)   4m9s
    cattle-system         cattle-cluster-agent-6f588568-zl55k                     0/1     CrashLoopBackOff   4 (29s ago)   3m53s
    kube-system           cloud-controller-manager-sempre1-ctrl                   1/1     Running            0             20m
    kube-system           cloud-controller-manager-sempre1-etcd                   1/1     Running            0             20m
    kube-system           etcd-sempre1-etcd                                       1/1     Running            0             19m
    kube-system           helm-install-rke2-calico-7dxlb                          0/1     Completed          2             20m
    kube-system           helm-install-rke2-calico-crd-wzffm                      0/1     Completed          0             20m
    kube-system           helm-install-rke2-coredns-zs9rl                         0/1     Completed          0             20m
    kube-system           helm-install-rke2-ingress-nginx-gtkv8                   0/1     CrashLoopBackOff   6 (40s ago)   20m
    kube-system           helm-install-rke2-metrics-server-blcf4                  0/1     CrashLoopBackOff   6 (51s ago)   20m
    kube-system           kube-apiserver-sempre1-ctrl                             1/1     Running            0             20m
    kube-system           kube-controller-manager-sempre1-ctrl                    1/1     Running            0             20m
    kube-system           kube-proxy-sempre1-ctrl                                 1/1     Running            0             20m
    kube-system           kube-proxy-sempre1-etcd                                 1/1     Running            0             20m
    kube-system           kube-proxy-sempre1-wrk1                                 1/1     Running            0             17m
    kube-system           kube-proxy-sempre1-wrk2                                 1/1     Running            0             17m
    kube-system           kube-scheduler-sempre1-ctrl                             1/1     Running            0             20m
    kube-system           rke2-coredns-rke2-coredns-58fd75f64b-kfb69              1/1     Running            0             19m
    kube-system           rke2-coredns-rke2-coredns-58fd75f64b-rzpsg              1/1     Running            0             20m
    kube-system           rke2-coredns-rke2-coredns-autoscaler-768bfc5985-hcf4b   1/1     Running            0             20m
    tigera-operator       tigera-operator-586758ccf7-rc9tq                        1/1     Running            0             19m
    
    Looking the logs seems cluster agent is unable to ping the rancher server, but if I do a curl on the same URL it responds with a pong.
    ERROR: <https://rancher75182.senode.dev/ping> is not accessible (Could not resolve host: rancher75182.senode.dev)
    
    helm pods report error as well:
    /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-ingress-nginx-gtkv8 -n cattle-system
    Error from server (NotFound): pods "helm-install-rke2-ingress-nginx-gtkv8" not found
    /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-metrics-server-blcf4 -n cattle-system
    Error from server (NotFound): pods "helm-install-rke2-metrics-server-blcf4" not found
    c
    • 2
    • 24
  • m

    millions-ocean-48249

    03/14/2023, 8:55 PM
    I am back with another issue I am running into... I have everything almost working with my rke2 vsphere cpi/csi enabled cluster, but I am having a problem with the windows node-driver-registrar container... Has anyone run into the following error before?
    I0314 12:43:17.862979    7024 main.go:109] "Kubelet registration probe created" path="\\var\\lib\\kubelet\\plugins\\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>\\registration"
    
    I0314 12:43:20.196288    7024 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix \\\\var\\\\lib\\\\kubelet\\\\plugins\\\\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>\\\\csi.sock: connect: A socket operation was attempted to an unreachable network.",}
    
    E0314 12:43:20.196288    7024 main.go:122] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix \\\\var\\\\lib\\\\kubelet\\\\plugins\\\\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>\\\\csi.sock: connect: A socket operation was attempted to an unreachable network.", restarting registration container.
    But I do see this does exist on the windows node:
    PS C:\var\lib\kubelet\plugins\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>> ls
    
    
        Directory: C:\var\lib\kubelet\plugins\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>
    
    
    Mode                LastWriteTime         Length Name
    ----                -------------         ------ ----
    -a----         3/8/2023   3:18 AM              0 csi.sock
  • h

    hundreds-evening-84071

    03/15/2023, 8:30 PM
    On RHEL 8 VMs... For RKE2, looking at this document, https://docs.rke2.io/install/configuration
    Method of installation to use. Default is on RPM-based systems
    rpm
    , all else
    tar
    And, with Rancher 2.7.1.... Does it make sense to create RKE2 cluster then import into Rancher or create RKE2 cluster from Rancher? Is one method better over other?
    g
    • 2
    • 8
  • a

    abundant-noon-17295

    03/16/2023, 6:24 AM
    does it make any sense to use private CA signed certs for “internal” comms, ie. kubelets <-> apiserver?
    c
    • 2
    • 2
  • h

    hundreds-evening-84071

    03/16/2023, 3:57 PM
    Creating RKE2 custom cluster from Rancher UI 2.7.1, and when I select Cloud Provider to External, I notice that Cloud Provider Config line opens up (screen shot below). If I leave that blank cluster creation does not work. TBH I am not sure what to put there? Is this where I can say "--disable-cloud-controller"?
    h
    • 2
    • 10
  • b

    broad-farmer-70498

    03/16/2023, 9:50 PM
    I'm trying to rebuild a node that was in a rke2 cluster. I've got it rebuilt and rejoined but it's in a notready state because
    container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
    but I have disabled deploying cni via rke2 (manual install of cilium). This is a chicken/egg situation that I'm not sure how I haven't run into it before now. How can I get kubelet to come up ready enough to get cni from daemonset in the cluster?
    c
    • 2
    • 33
  • f

    flaky-jackal-94200

    03/17/2023, 12:06 PM
    Hello, does anyone know if it is possible to change the default pod/container logs directories from
    /var/log/pods
    and
    /var/log/containers
    to something else? Setting
    --log-dir
    in the kubelet is deprecated unfortunately. Also, I'm not looking for redirection, I need
    /var/log/pods
    and
    /var/log/containers
    to have no logs from kubernetes.
  • e

    echoing-tomato-53055

    03/17/2023, 5:26 PM
    @here: is anyone faced/facing the below issue when spinning up kubernetes cluster using rancher 2.7.
    level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/
    c
    • 2
    • 8
  • g

    gorgeous-receptionist-16083

    03/18/2023, 4:15 AM
    Hello so I’m quite new to running kubernetes, I have ran it in a test environment and created some apps. However I’m looking to run it in production. How much work is it to maintain a cluster, meaning is it something that requires on going maintenance, or can I just leave it and it will keep running. I know that certs need to be updated for the control plane , is that something that is done automatically with rke2 I think with installation with kubeadm it is not. I really like rke2 I’m just wondering how much operational maintenance is involved. Thanks in advance
    h
    • 2
    • 1
  • a

    ancient-army-24563

    03/18/2023, 6:18 AM
    I have cluster stuck with waitibg for pboes : kubelet
  • a

    ancient-army-24563

    03/18/2023, 6:18 AM
    can someone helpk with this issue
  • w

    worried-ram-81084

    03/20/2023, 6:31 AM
    after launching the master, how to edit the node yaml file to include the external ip?
  • h

    hundreds-evening-84071

    03/20/2023, 3:47 PM
    After deploying RKE2 cluster via Rancher 2.7.1... I want to change "Cloud Provider to External" I can do that by edit Yaml and add
    cloud-provider-name: external
    under
    machineSelectorConfig
    . This appears to work... My question with this is, do I also need to include
    disable-cloud-controller: true
    ? If so, where does this get added in YAML or the GUI?
    h
    • 2
    • 1
  • r

    rapid-scientist-25800

    03/22/2023, 11:57 AM
    Hi, is rke2 considered stable to use as a basis to install rancher manager on ?
Powered by Linen
Title
r

rapid-scientist-25800

03/22/2023, 11:57 AM
Hi, is rke2 considered stable to use as a basis to install rancher manager on ?
View count: 1