rancher-users #rke2

rke2

stale-painting-80203

05/17/2022, 2:33 PM

I am trying to install and run rke2 in a SLES 15 SP3 VM and facing issues. I am using the guide here https://docs.rke2.io/install/quickstart/. I have disabled the firewall and enabled IPv4/IPv6 port forwarding in Wicked. Seems the server is running, but I do see errors in the logs. Also when I installed the agent in the same VM I got an error:

msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:48636->127.0.0.1:6444: read: connection reset by peer

Curl gives following error:

curl -v <https://127.0.0.1:6444/cacerts>
*  Trying 127.0.0.1:6444...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:6444 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:6444

Anyone know what this might be caused by? curl -k https://127.0.0.1:6443 gives a response.

rke2-server.service - Rancher Kubernetes Engine v2 (server)
   Loaded: loaded (/etc/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-05-16 20:46:15 PDT; 4min 55s ago
    Docs: <https://github.com/rancher/rke2#readme>
  Process: 48957 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited>
  Process: 48960 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 48961 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Main PID: 48962 (rke2)
   Tasks: 205
   CGroup: /system.slice/rke2-server.service
       ├─ 2652 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2673 /pause
       ├─ 2707 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2711 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2737 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2758 /pause
       ├─ 2778 /pause
       ├─ 2813 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2836 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2841 /pause
       ├─ 2866 /pause
       ├─ 2882 kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wa>
       ├─ 3069 etcd --config-file=/var/lib/rancher/rke2/server/db/etcd/config
       ├─ 3761 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3782 /pause
       ├─ 3807 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3835 /pause
       ├─ 3863 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3890 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3920 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3940 /pause
       ├─ 3941 /pause
       ├─ 4073 /coredns -conf /etc/coredns/Corefile
       ├─38373 kube-apiserver --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --allow-privileged>
       ├─42523 /cluster-proportional-autoscaler --namespace=kube-system --configmap=rke2-coredns-rke2-coredns-aut>
       ├─43471 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─43493 /pause
       ├─43738 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
       ├─45408 /usr/sbin/runsvdir -P /etc/service/enabled
       ├─45734 runsv felix
       ├─45735 runsv monitor-addresses
       ├─45736 runsv allocate-tunnel-addrs
       ├─45737 runsv node-status-reporter
       ├─45738 runsv cni
       ├─45739 calico-node -monitor-addresses
       ├─45740 calico-node -felix
       ├─45741 calico-node -allocate-tunnel-addrs
       ├─45742 calico-node -status-reporter
       ├─45743 calico-node -monitor-token
       ├─46226 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─46261 /pause
       ├─48962 /opt/rke2/bin/rke2 server
       ├─48995 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/contai>
       ├─49051 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-freque>
       └─49452 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server>

May 16 20:50:57 suse-vm rke2[48962]: E0516 20:50:57.578889  48962 memcache.go:101] couldn't get resource list for metr>
May 16 20:51:05 suse-vm rke2[48962]: I0516 20:51:04.992233  48962 trace.go:205] Trace[411526421]: "Reflector ListAndWa>
May 16 20:51:05 suse-vm rke2[48962]: Trace[411526421]: ---"Objects listed" 61358ms (20:51:03.401)
May 16 20:51:05 suse-vm rke2[48962]: Trace[411526421]: [1m2.51858886s] [1m2.51858886s] END
May 16 20:51:08 suse-vm rke2[48962]: time="2022-05-16T20:51:08-07:00" level=info msg="Event(v1.ObjectReference{Kind:\"A>
May 16 20:51:09 suse-vm rke2[48962]: E0516 20:51:09.360628  48962 memcache.go:196] couldn't get resource list for metr>
May 16 20:51:09 suse-vm rke2[48962]: E0516 20:51:09.931690  48962 memcache.go:101] couldn't get resource list for metr>
May 16 20:51:10 suse-vm rke2[48962]: time="2022-05-16T20:51:10-07:00" level=info msg="Starting /v1, Kind=Secret control>
May 16 20:51:10 suse-vm rke2[48962]: time="2022-05-16T20:51:10-07:00" level=info msg="Starting /v1, Kind=Node controlle>
May 16 20:51:10 suse-vm rke2[48962]: I0516 20:51:10.690858  48962 leaderelection.go:248] attempting to acquire leader

straight-fireman-71417

05/17/2022, 5:49 PM

Is there anyway to perform an RKE2 snapshot and upload it to 2 buckets?

magnificent-vr-88571

05/18/2022, 10:08 AM

I am using rke2 v1.21.5 and facing issue in using private registry.

rke2 --version
rke2 version v1.21.5+rke2r2 (9e4acdc6018ae74c36523c99af25ab861f3884da)
go version go1.16.6b7

I have created registries.yaml with aws ECR as private registry

mirrors:
  <http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
    endpoint:
      - "<https://2341234234.dkr.ecr.yyyyy.amazonaws.com>"
configs:
  "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
    auth:
      username: xxxxx
      password: xxxxx
    tls:
      insecure_skip_verify: true

While I create a pod to pull registry with following image name in pod.yaml, it fails with image pull error.

image: <http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>

Looking forward some insights

victorious-analyst-3332

05/18/2022, 2:03 PM

Howdy all 👋 Is it expected that the

kube-controller-manager

metrics on port 10252 are not exposed in RKE2 deployments. We are currently testing

v1.22.9+rke2r2

deployed via Rancher

v2.6.5

and are seeing the following differences from RKE deployments. Thanks a lot. RKE cluster:

# netstat -tulpn | grep kube-contro
tcp6       0      0 :::10252                :::*                    LISTEN      17396/kube-controll
tcp6       0      0 :::10257                :::*                    LISTEN      17396/kube-controll

RKE2 cluster:

# netstat -tulpn | grep kube-contro
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      39961/kube-controll

gifted-cricket-25537

05/19/2022, 10:54 AM

Hi all! I'm probably blind, but I couldn't find the solution to this problem yet. I'm trying to start my cluster with:

cni: "calico"
disable-kube-proxy: true

and since there's no

kube-proxy

installed the

tigera-operator

can't reach the API

2022/05/19 07:50:02 [ERROR] Get "<https://10.43.0.1:443/api?timeout=32s>": dial tcp 10.43.0.1:443: i/o timeout

I know that I could override it using the

kubernetes-services-endpoint

CM, although I can't inject that during the installation. Any clues?

faint-airport-83518

05/19/2022, 11:17 PM

with rke2 we don't need leader election anymore? https://github.com/rancher/rke2/issues/349. I tried pulling the logic out from https://github.com/rancherfederal/rke2-azure-tf/blob/main/modules/custom_data/files/rke2-init.sh#L51-L71 and am getting an error.

curved-caravan-26314

05/20/2022, 12:22 AM

Hello, I'm reading through the rke2 ha documentation. it doesn't talk about the need to add kubectl again for the other server nodes. Is it safe to still install kubectl on the other server nodes?

alert-oxygen-95787

05/20/2022, 10:40 AM

Hi, I'm a k3s user - just for local testing, my company uses EKS in production. Do Rancher recommend using k3s or rke2 for local testing? Is work on k3s eventually going to stop in preference to rke2? TIA

👀 1

curved-caravan-26314

05/20/2022, 12:05 PM

Morning, I am looking to go from using ubuntu to Suse Leap 15.3. For the hardware I have, the legacy install of ubuntu is what I use. What is the equivalent to install suse leap?

shy-zebra-53074

05/20/2022, 10:39 PM

Happy Friday! I’m performing an analysis against RKE2 using the DISA STIGs and I came across this one:

Rule Title: The Kubernetes Controller Manager must use TLS 1.2, at a minimum, to protect the confidentiality of sensitive data during electronic dissemination.

Discussion: The Kubernetes Controller Manager will prohibit the use of SSL and unauthorized versions of TLS protocols to properly secure communication.

The use of unsupported protocol exposes vulnerabilities to the Kubernetes by rogue traffic interceptions, man-in-the-middle attacks, and impersonation of users or services from the container platform runtime, registry, and key store. To enable the minimum version of TLS to be used by the Kubernetes Controller Manager, the setting "tls-min-version" must be set.

Check Text: Change to the /etc/kubernetes/manifests/ directory on the Kubernetes Master Node. Run the command:

grep -i tls-min-version * 

If the setting "tls-min-version" is not configured in the Kubernetes Controller Manager manifest file or it is set to "VersionTLS10" or "VersionTLS11", this is a finding.

Fix Text: Edit the Kubernetes Controller Manager manifest file in the /etc/kubernetes/manifests directory on the Kubernetes Master Node. Set the value of "--tls-min-version" to "VersionTLS12" or higher.

However I don’t see the

tls-min-version

flag set for the

kube-controller-manager

service:

kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --cloud-provider=aws --cloud-config= --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.0.0.0/12 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --use-service-account-credentials=true

thankful-plastic-94519

05/22/2022, 1:08 AM

I've searched and found a general "no" to this question, but I wonder more specifically about the reasoning / why. I have a busy cluster with high churn rate, my current issues stem from IO latency in the etcd space / control plane. Because rke2 only supports colocated etcd & control plane components, this seems like a pretty severe shortcoming. Is there any advise towards addressing this type of issue with rke2?

magnificent-vr-88571

05/23/2022, 12:49 PM

@creamy-pencil-82913 @gray-lawyer-73831 In reference to https://rancher-users.slack.com/archives/C01PHNP149L/p1652868532770339 Currently facing below error in pod events after setting up private registry with endpoint.

Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  21s                     default-scheduler  Successfully assigned default/ecr-pod to sv11
  Normal   BackOff    8s                      kubelet            Back-off pulling image "private-registry/utils-img:latest"
  Warning  Failed     8s                      kubelet            Error: ImagePullBackOff
  Normal   Pulling    <invalid> (x2 over 8s)  kubelet            Pulling image "private-registry/utils-img:latest"
  Warning  Failed     <invalid> (x2 over 8s)  kubelet            Failed to pull image "private-registry/utils-img:latest": rpc error: code = NotFound desc = failed to pull and unpack image "private-registry/utils-img:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found  
Warning  Failed     <invalid> (x2 over 8s)  kubelet            Error: ErrImagePull

I am trying to add rewrite as mentioned https://github.com/k3s-io/k3s/issues/5502#issuecomment-1109107238 to overcome this issue. After adding rewrite

/etc/rancher/rke2/registries.yaml

its not reflected in the environment, would like to know rewrite in

/var/lib/rancher/rke2/agent/etc/containerd/config.toml

to overcome this issue. correct me if am wrong.

ambitious-butcher-83703

05/23/2022, 11:39 PM

Hello — Using RKE2 in a multinode air-gapped setup. If I have some custom containers I want to able to run in the cluster, do I need to have the tar files in /var/lib/rancher/rke2/agent/images/ in each of the nodes? or just on the master. Will the pods only be able to pull the image if it exists on the node its trying to run on? Thanks in advance!

billions-kite-9416

05/25/2022, 10:34 AM

Hi everyone, I am having some trouble creating a custom rke2 cluster from Rancher UI and the error message is not really helpful. The rancher-system-agent binary gets downloaded to the nodes but there is an error saying

run.sh

is not found

May 25 13:18:53  rancher-system-agent[4497]: time="2022-05-25T13:18:53+03:00" level=info msg="Pulling image <http://index.docker.io/rancher/system-a|index.docker.io/rancher/system-a>
gent-installer-rke2:v1.23.6-rke2r2"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[48304be41fe2284adc2b5937c9d5c9c7b09009b2cae77
6320036286c2f0c3191_0:stderr]: sh: 1: run.sh: not found"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Command sh [-c run.sh] finished
with err: <nil> and exit code: 127"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=error msg="error executing instruction 0: <nil>"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=error msg="error encountered during parsing of last run
time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] No image provided, creating empt
y working directory /var/lib/rancher/agent/work/20220525-131853/48304be41fe2284adc2b5937c9d5c9c7b09009b2cae776320036286c2f0c3191_0"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Running command: sh [-c rke2 etc
d-snapshot list --etcd-s3=false 2>/dev/null]"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapsho
t list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 127"
May 25 13:18:56  rancher-system-agent[4497]: time="2022-05-25T13:18:56+03:00" level=error msg="error loading x509 client cert/key for probe
kube-apiserver (/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt//var/lib/rancher/rke2/server/tls/client-kube-apiserver.key): open /var/lib/
rancher/rke2/server/tls/client-kube-apiserver.crt: no such file or directory"

👀 1

future-monitor-61871

05/25/2022, 7:39 PM

Is there a fully populated config.yaml out there somewhere that I can look at? Or is there an option to produce a blank one from the rke2 executable? I've dug on github but haven't found such an artifact.

hundreds-evening-84071

05/26/2022, 12:50 PM

When creating RKE2 cluster on Linux, I have /var/lib/rancher separate partition So, wondering what directory is used when a windows worker node is added for RKE2? I also want to create a separate partition there.

shy-zebra-53074

05/26/2022, 4:35 PM

Hey all! I’ve been able to set custom configs via:

kube-apiserver-arg

kube-controller-manager-arg

and

kube-scheduler-arg

However, I would like to ensure that a certain config is NOT set that is currently being set by default. For example I would like to unset the

--hostname-override

flag for the

kubelet

service and make sure that flag is not passed when starting the kubelet service

bored-rain-98291

05/27/2022, 6:18 PM

Greetings: I am configuring a cluster using RKE2. I understand that nodes are for the control plane and that agents are for running application workloads. Is there any best-practice advice on mixing/matching on the same node? also what would the memory requirements be? I have only 3 nodes which have blocks of memory that i can adjust. Currently they have 4gb and this is for a dev cluster. Interested in the optimal setup for this small size? 1 node/2 agents, 2 nodes - 1 agent? thanks for any insight!

great-photographer-94826

05/30/2022, 8:24 AM

Hi #rke2! I want to create a high-availability RKE2 cluster where it is necessary to distribute nodes at two sites (e.g. EU-A and EU-B geographically distributed sites). It is recommended to have an odd number of masters in a cluster. But it is necessary for each sites to be failure tolerance. Is it good idea when I place 3 master nodes each sites (EU-A 3 master nodes, EU-B 3 master nodes)? Then fault tolerance is ensured both within and between sites. Am I wrong? I don't want to build multi-cluster RKE2 architecture because of Longhorn.

billowy-animal-38808

05/31/2022, 2:39 PM

Hello everyone. Does Rke2 allow the creation of LoadBalancer services by default? I have a rke2 Cluster and I created a LoadBalancer service but when listing the services it is in a

pending

state. Can someone explain to me what to do to enable this option?

sparse-businessperson-74827

06/01/2022, 1:28 PM

Hi, I am having problem with one of the nodes joining rke2 cluster running Rancher 2.6.2 The server has died and was rebuilt. The references to old server were removed from UI. Now when I try to add the node(same IP and name as before) I am getting this:

Jun  1 15:20:33 node-1 rke2[35837]: time="2022-06-01T15:20:33+02:00" level=error msg="Failed to connect to proxy" error="unexpected EOF"

Jun  1 15:20:33 node-1 rke2[35837]: time="2022-06-01T15:20:33+02:00" level=error msg="Remotedialer proxy error" error="unexpected EOF"

bored-rain-98291

06/02/2022, 1:14 PM

Greetings friends. I am trying to install RKE2 on 3 centos8 servers. When i configure the first node with the config.yaml in /etc/rancher/config.yaml it causes the node to crash and not start clean. Ive tried to get the system to recognize the other server nodes but simply cant get it to work. I can get it to work with a single node. Any help would be greatly appreciated.

narrow-noon-75604

06/02/2022, 3:47 PM

Hi, We are trying to deploy the RKE2 on a centos7 cluster...containing 1 server node and 3 agent nodes...The complete installation was smooth and able to get all the nodes in "Ready" state. But the issue is that the pods under "calico-system" namespace are going into "CrashLoopBackOff" with "Authorization error" as below,

$ kubectl logs -f pod/calico-node-r2wj8 -n calico-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy)

Created a new clusterrolebinding with "system:kubelet-api-admin" role but that did not help. Please help us with the steps to fix this issue and let me know if you need any more details

salmon-dress-41836

06/07/2022, 5:38 AM

There are server flags provided to modify some of the items in the generated static pod manifests but is there any way to modify anything outside of that? E.g. liveness probes

narrow-noon-75604

06/07/2022, 2:28 PM

Hi, I want to deploy rke2 with dual stack but could not able to find any references for config.yaml. can anyone please share me any reference links of config.yaml for dual stack configuration using calico network. This is the current configuration I am using,

write-kubeconfig-mode: "0644"
tls-san:
  - "<http://master.167.254.204.58.nip.io|master.167.254.204.58.nip.io>"
node-label:
  - "nodetype=master"
cluster-cidr: "10.42.0.0/16"
service-cidr: "10.43.0.0/16"
cluster-dns: "10.43.0.10"
cluster-domain: "<http://master.xxx.xxx.xxx.xxx.nip.io|master.xxx.xxx.xxx.xxx.nip.io>"
cni:
  - calico
disable:
  - rke2-canal
  - rke2-kube-proxy

curved-caravan-26314

06/07/2022, 4:30 PM

Is OEL 7.9 compatible with rke2

hundreds-hairdresser-46043

06/08/2022, 9:22 AM

Hi People. So firewallD has issues with the CNI plugins as per this: https://docs.rke2.io/known_issues/#firewalld-conflicts-with-default-networking my question would be then - what options do I have to enable a firewall on the servers themselves ? so far we just left the firewalls off - but at some point we would need to enable them - suggestions?

hundreds-evening-84071

06/08/2022, 5:32 PM

hey guys, running rancher 2.6.5 stable release... created RKE2 cluster and also added a windows worker node... however, I see by default, it uses

c:\var\lib\rancher

as directory... is there an option - or a way I can specify something other than c-drive ? perhaps like

d:\var\lib\rancher

? reason I want to do this is to keep users-stuff off the c-drive to prevent c-drive from filling up and crashing the OS.

narrow-noon-75604

06/08/2022, 7:18 PM

Hi, I am trying to install RKE2 v1.23.6 on a RHEL8 cluster (1server & 4agents) using the Airgap tarball method. The installation is successful, the nodes are added properly and into Ready state.

kubectl get nodes
NAME           STATUS                     ROLES                       AGE    VERSION
rke2-master    Ready,SchedulingDisabled   control-plane,etcd,master   112m   v1.23.6+rke2r2
rke2-worker1   Ready                      <none>                      108m   v1.23.6+rke2r2
rke2-worker2   Ready                      <none>                      108m   v1.23.6+rke2r2
rke2-worker3   Ready                      <none>                      108m   v1.23.6+rke2r2

All the pods under "kube-system" & "calico-system" namespace are up & Running properly.

kubectl get po -n calico-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-69cb5d9c7b-xllwx   1/1     Running   0          113m
calico-node-86pcd                          1/1     Running   0          110m
calico-node-dxtjk                          1/1     Running   0          110m
calico-node-lb8n7                          1/1     Running   0          113m
calico-node-ld8mf                          1/1     Running   0          110m
calico-typha-86c8b747c4-rqfhk              1/1     Running   0          113m
calico-typha-86c8b747c4-xs7zs              1/1     Running   0          110m

kubectl get po -n kube-system
NAME                                                    READY   STATUS      RESTARTS   AGE
cloud-controller-manager-rke2-master                    1/1     Running     0          114m
etcd-rke2-master                                        1/1     Running     0          114m
helm-install-rke2-calico-crd-j44qb                      0/1     Completed   0          114m
helm-install-rke2-calico-hv4xg                          0/1     Completed   1          114m
helm-install-rke2-coredns-btsgx                         0/1     Completed   0          114m
helm-install-rke2-ingress-nginx-c99sw                   0/1     Completed   0          114m
helm-install-rke2-metrics-server-j5gjq                  0/1     Completed   0          114m
kube-apiserver-rke2-master                              1/1     Running     0          113m
kube-controller-manager-rke2-master                     1/1     Running     0          114m
kube-proxy-rke2-master                                  1/1     Running     0          114m
kube-proxy-rke2-worker1                                 1/1     Running     0          110m
kube-proxy-rke2-worker2                                 1/1     Running     0          110m
kube-proxy-rke2-worker3                                 1/1     Running     0          110m
kube-scheduler-rke2-master                              1/1     Running     0          114m
rke2-coredns-rke2-coredns-69c8f974c-gvwqg               1/1     Running     0          7m56s
rke2-coredns-rke2-coredns-69c8f974c-qmq8m               1/1     Running     0          7m56s
rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-4g4sw   1/1     Running     0          114m
rke2-ingress-nginx-controller-2bhbc                     1/1     Running     0          113m
rke2-ingress-nginx-controller-gnpk5                     1/1     Running     0          110m
rke2-ingress-nginx-controller-znh8z                     1/1     Running     0          110m
rke2-ingress-nginx-controller-ztpjf                     1/1     Running     0          110m
rke2-metrics-server-6564db4569-m5kcc                    1/1     Running     0          113m

I have deployed kafka but the pods are going to "CrashLoopBackOff" with a dns error,

java.net.UnknownHostException: zookeeper.msgbus.svc: Temporary failure in name resolution

The logs of coredns pods are throwing a lot of errors as,

[ERROR] plugin/errors: 2 4686730224998678655.5699833978585684595. HINFO: read udp xx.xx.xx.xx:51070->yy.yy.yy.yy:53: read: no route to host
[ERROR] plugin/errors: 2 4686730224998678655.5699833978585684595. HINFO: read udp xx.xx.xx.xx:42332->yy.yy.yy.yy:53: read: no route to host
[ERROR] plugin/errors: 2 4686730224998678655.5699833978585684595. HINFO: read udp xx.xx.xx.xx:57092->yy.yy.yy.yy:53: read: no route to host

I have opened port 53 to allow both tcp and udp traffic and am not about the cause of this issue. Any suggestions would be appreciated and let me know if you need any more debug logs

brief-mouse-13981

06/09/2022, 5:59 AM

Hi all. I loved rke2 for a every long time and know i am planning to implement it on a customers site. But the customer thinks somehow that the vanilla Kubernetes has more of a stable development team (because of the fact that it is vanilla). Can anybody help me with reasonable arguments so i can convince the customer to choose for rke2 ?

Title

brief-mouse-13981

06/09/2022, 5:59 AM

creamy-pencil-82913

06/09/2022, 6:32 AM

If you reach out to SUSE sales they should be able to help you with that. They have some good material. Unless you're planning on going without support, I guess :)

brief-mouse-13981

06/09/2022, 6:42 AM

jup, support isn´t something the customer is willing to pay for. I will try to scrape some material from the sales website from suse. Thanks

rapid-helmet-86074

06/09/2022, 1:51 PM

Over a decade ago, I installed a bunch of Linux VMs for various projects that I was maintaining but was told I'd be ramped down. At that time the three Linux distros with support were RHEL, SUSE, & Ubuntu. RHEL & SUSE required you install the support-bearing distro immediately or you'd never have support, but Ubuntu allowed same install and people could just call up for a year of support later if desired. That's why I chose Ubuntu for that customer letting them know that they had that option if they wanted/needed it later after I'd moved on. I didn't ask their sales folks if an unsupported cluster could have support turned on, but I did ask if I wanted to start out getting support the first year for both prod & dev clusters and then drop dev support after the first year and it was up and running and the sales guy said that was pretty common. I also appreciated with Rancher versus, say OpenShift, that the free version of the software can be installed without a license cost (and thus without vendor support) at development teams' shops or even more often in small clusters to prototype a feature (I currently have four small segmented cluster for different Rancher installs for some testing, $0 license fees). Another argument for a "Kubernetes distribution" such as RKE2 with Rancher, over vanilla Kubernetes is that you get a lot of bundled useful support tools that you'd otherwise have to install piece by piece. If the distribution mismatches the desired toolset, then it's not a help, but it can be worth it to find such things first. Next argument, and it's a balance, is that a distribution like Rancher & RKE2 will have integration testing for the various bundled parts. This will lead to sometimes those parts not being the very latest versions (even usually), but it also usually means less problems with them that have to be solved on the user's cluster than when you have vanilla weaving everything together. So it's a bit of a pro and a con. Final thing that RKE2 specifically gives you is a more locked down default profile than you'll get from vanilla Kubernetes. At the same time the vanilla Kubernetes docs told me I had to disable SELinux I could, and eventually did, install RKE2 with SELinux enforcing. So it's saving some work on lockdowns and security compliance checklists for you as well.

brief-mouse-13981

06/09/2022, 2:23 PM

@rapid-helmet-86074 thank you for your answer. I think i will just install RKE2 with these arguments. That should suffice.

careful-table-97595

06/23/2022, 1:48 AM

@brief-mouse-13981 And if you want something really nice and near maintenance-free, you can couple rke2 with flatcar OS (https://flatcar-linux.org/) to get an immutable OS as CoreOS is for Openshift, then add the flatcar upgrade operator (https://github.com/flatcar-linux/flatcar-linux-update-operator) to automatically maintain your OS up-to-date from a kubernetes-aware mecanism, and also the rancher system-upgrade-controller to upgrade rke2 as a workload (https://github.com/rancher/system-upgrade-controller, also mentioned here (https://docs.rke2.io/upgrade/automated_upgrade/) Nothing looking like a kubernetes cluster for developers only! 🙂

That said, I'm still working on making automated rke2 upgrades work. The rest, however, is a breeze!

View count: 25