green-winter-88935
06/10/2022, 8:06 AMrapid-helmet-86074
06/10/2022, 4:41 PMalert-potato-16010
06/13/2022, 11:58 AMripe-queen-73614
06/13/2022, 2:53 PMfuture-monitor-61871
06/13/2022, 7:32 PMbusy-crowd-80458
06/14/2022, 7:28 AMkind-air-74358
06/14/2022, 9:22 AMvictorious-ambulance-564
06/15/2022, 9:04 AMbored-rain-98291
06/15/2022, 4:53 PMrapid-helmet-86074
06/16/2022, 5:47 PM--default-ssl-certificate
argument to rke2-ingress-nginx-controller
config to spec.template.spec.containers.args array as an append so that all the default arguments are still there without me specifying (as I've noticed they change between versions at times).
I know I can replace a value, but it isn't clear if I can specify an array append to what's there in a generic fashion.billions-easter-91774
06/16/2022, 7:17 PM"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
When i try to debug containerd/container with ctr or crictrl i always get that there is no containerd.sock (i can only find
containerd.sock.ttrpc
There was a weird issue on 2 ctrl planes: something about not enough filedescriptors. Unfortuna i was not able to see who tried to open too many files, but i'im not sure if this is not more of an containerd issue who starts 2 etcd container constantly
My google and debug magic is gone. Any ideas/suggestions?hallowed-hair-79157
06/17/2022, 12:54 AMJun 16 21:53:06 test-rke-server-1 rke2[7641]: F0616 21:53:06.418489 7641 csi_plugin.go:305] Failed to initialize CSINode after retrying: timed out waiting for the condition
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: goroutine 796 [running]:
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.stacks(0xc000a0e101|k8s.io/klog/v2.stacks(0xc000a0e101>, 0xc0002122c0, 0x82, 0x291)
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.(*loggingT).output(0x7648520|k8s.io/klog/v2.(*loggingT).output(0x7648520>, 0xc000000003, 0x0, 0x0, 0xc000586540, 0x0, 0x618e831, 0xd, 0x131, 0x0)
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x1e5
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.(*loggingT).printf(0x7648520|k8s.io/klog/v2.(*loggingT).printf(0x7648520>, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x4efe5d9, 0x2f, 0xc0016f2cb0, 0x1, ...)
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:753 +0x19a
Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.Fatalf(...)|k8s.io/klog/v2.Fatalf(...)>
(error trace goes for hundreds of lines)
has anyone run into this issue before/know what is wrong? was not able to find much from googling. I am running k8s/rke2 version 1.22.10+rke2r2 on centos 8 stream
note that this only happens on the second server node, my first node is fine:
NAME STATUS ROLES AGE VERSION
test-rke-server-0 Ready control-plane,etcd,master 4h44m v1.22.10+rke2r2
silly-jordan-81965
06/17/2022, 8:24 AM--kube-controller-manager-arg value
--kube-scheduler-arg value
faint-airport-83518
06/17/2022, 5:15 PMmagnificent-vr-88571
06/17/2022, 7:45 PMmagnificent-vr-88571
06/18/2022, 10:21 PM$ kubectl version
Client Version: version.Info{Major:“1”, Minor:“23", GitVersion:“v1.23.6”, GitCommit:“ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:“clean”, BuildDate“2022 04 14T0849:13Z”, GoVersion:“go1.17.9”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1", Minor:“21”, GitVersion:“v1.21.5+rke2r2", GitCommit:“aea7bbadd2fc0cd689de94a54e5b7b758869d691”, GitTreeState:“clean”, BuildDate“2021 10 04T2239:02Z”, GoVersion:“go1.16.7b7", Compiler:“gc”, Platform:“linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
server1 Ready <none> 10d v1.22.9+rke2r2
server2 Ready <none> 10d v1.21.5+rke2r2
server3 Ready,SchedulingDisabled control-plane,etcd,master 58d v1.21.5+rke2r2I did following steps. 1. Drained server1, 2. rke2-killall.sh in server1 3. Replaced ‘/usr/local/bin/rke2*’ in server1 with server2 binaries 4. started agent in server1 “systemctl start rke2-agent” it became like following.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
server1 Ready <none> 10d v1.21.5+rke2r2
server2 Ready <none> 10d v1.21.5+rke2r2
server3 Ready,SchedulingDisabled control-plane,etcd,master 58d v1.21.5+rke2r2Still it shows client version mismatch. would like to know how to update this Client version mismatch.
$ kubectl version
Client Version: version.Info{Major:“1", Minor:“23”, GitVersion:“v1.23.6", GitCommit:“ad3338546da947756e8a88aa6822e9c11e7eac22”, GitTreeState:“clean”, BuildDate“2022 04 14T0849:13Z”, GoVersion:“go1.17.9", Compiler:“gc”, Platform:“linux/amd64"}
Server Version: version.Info{Major:“1”, Minor:“21", GitVersion:“v1.21.5+rke2r2”, GitCommit:“aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:“clean”, BuildDate“2021 10 04T2239:02Z”, GoVersion:“go1.16.7b7”, Compiler:“gc”, Platform:“linux/amd64”}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1
curved-caravan-26314
06/19/2022, 12:09 PMeager-refrigerator-66976
06/20/2022, 2:33 PMaws-eni-cni
and having interesting issue, the bootstrap controlplane node starts-up fine but other nodes are failing as they get wrong "server": "<https://IP:9345>",
that IP
isn’t the node IP and I have no idea how it was discovered… I did check I have no pods running with such IP on bootstrap node… however that IP is one of the EC2 instance secondary IP addresses… any idea how can I fix this? 🙏curved-caravan-26314
06/20/2022, 3:49 PMcurved-caravan-26314
06/20/2022, 4:41 PMsudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
systemctl stop ufw
systemctl disable ufw
create file /etc/sysctl.d/90-rke2.conf
net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
modify /etc/resolv.conf
nameserver from 127.0.0.53 to 8.8.8.8
search from <http://attlocal.net|attlocal.net> to <http://my.domain.com|my.domain.com>
bland-jackal-22983
06/21/2022, 5:34 AMps
before it become unresponsive:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
833 root 20 0 948732 156860 89276 S 100.3 0.2 0:17.59 rke2
1426 root 20 0 1045476 317188 74388 S 70.4 0.5 0:02.39 kube-apiserver
1348 root 20 0 10.7g 62184 24912 S 10.3 0.1 0:00.41 etcd
992 root 20 0 830948 95468 64436 S 2.3 0.1 0:00.38 kubelet
1489 root 20 0 758424 50276 35488 S 1.0 0.1 0:00.50 kube-scheduler
973 root 20 0 766448 58876 38472 S 0.7 0.1 0:00.34 containerd
1508 root 20 0 752084 35188 27144 S 0.7 0.1 0:00.05 kube-proxy
after a while, i was logged off from the ssh connection: client_loop: send disconnect: Broken pipe
the server spec looks fine: each server has 6 physical cores, 64gb memory
os: ubuntu 22.04
rke2 version: latest stable
anyone knows what could be wrong? happy to provider any infocurved-caravan-26314
06/21/2022, 8:17 PMNAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-76d44b459c-qfdgq 1/1 Running 0 4m26s
cert-manager cert-manager-cainjector-9b679cc6-rnzcc 1/1 Running 0 4m26s
cert-manager cert-manager-webhook-57c994b6b9-r7cxs 1/1 Running 0 4m26s
kube-system cloud-controller-manager-rke2 1/1 Running 0 18m
kube-system etcd-rke2 1/1 Running 0 17m
kube-system helm-install-rke2-canal-zvs8x 0/1 Completed 0 17m
kube-system helm-install-rke2-coredns-frgcz 0/1 Completed 0 17m
kube-system helm-install-rke2-ingress-nginx-8mvg9 0/1 Completed 0 17m
kube-system helm-install-rke2-metrics-server-rp2jz 0/1 Completed 0 17m
kube-system kube-apiserver-rke2 1/1 Running 0 17m
kube-system kube-controller-manager-rke2 1/1 Running 0 18m
kube-system kube-proxy-rke2 1/1 Running 0 18m
kube-system kube-scheduler-rke2 1/1 Running 0 18m
kube-system rke2-canal-nkrmd 2/2 Running 0 16m
kube-system rke2-coredns-rke2-coredns-547d5499cb-4g4jj 1/1 Running 0 16m
kube-system rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-m2d8d 1/1 Running 0 16m
kube-system rke2-ingress-nginx-controller-s977x 1/1 Running 0 11m
kube-system rke2-metrics-server-6564db4569-9h8mk 1/1 Running 0 13m
bland-jackal-22983
06/23/2022, 2:28 AMjournalctl -u rke2-server -f
in the thread.
os: ubuntu 22.04
rke2 version: latest stablebland-jackal-22983
06/24/2022, 3:22 PMnarrow-noon-75604
06/27/2022, 2:30 PMJun 27 10:17:43 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[38318]: time="2022-06-27T10:17:43-04:00" level=fatal msg="starting kubernetes: preparing server: failed to validate server configuration: critical configuration value mismatch"
First Server configuration:
# BEGIN Adding RKE2 configuration
write-kubeconfig-mode: "0644"
tls-san:
- "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
node-label:
- "nodetype=master"
node-ip: "xxx.xxx.xxx.42,xxxx:xxx:x:xxx:xxx:xxxx:xxxx:aae"
cluster-cidr: "10.42.0.0/16,2001:cafe:42:0::/56"
service-cidr: "10.43.0.0/16,2001:cafe:42:1::/112"
cluster-dns: "10.43.0.10"
cluster-domain: "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
cni:
- calico
disable:
- rke2-canal
- rke2-kube-proxy
# END Adding RKE2 configuration
Second Server Configuration:
# BEGIN Adding RKE2 configuration
server: "<https://rke2-master1.xxx.xxx.xxx.42.nip.io:9345>"
token: "K10d463a80c8c1323f30fa6d97fcf91992454a43dc5c544f1c9a0de706b733b51ee::server:f6fd26cafff902300ba021b29b11eddc"
tls-san:
- "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
node-ip: "xxx.xxx.xxx.43,xxxx:xxx:x:xxx:xxx:xxxx:xxxx:5245"
cni:
- calico
disable:
- rke2-canal
- rke2-kube-proxy
# END Adding RKE2 configuration
There is no firewalld or iptables running on any of the nodes...Please help me in finding the issue herebored-rain-98291
06/27/2022, 2:39 PMbored-rain-98291
06/28/2022, 4:35 PMgreat-flag-38820
06/29/2022, 1:24 AMAs of v1.21.2, RKE2 supports selecting a different CNI via theI want to understand why RKE2 supports these different CNIs, but doesn't recompile them for FIPS compliance. Doesn't that go against RKE2's ethos of having a fully conformant distribution for US Government sector customers? Are there any options for people that want to use something like Multus, Calico (Enterprise) or Cilium but need all encryption to be FIPS validated?flag and comes bundled with several CNIs including Canal (default), Calico, Cilium, and Multus. Of these, only Canal (the default) is rebuilt for FIPS compliance.--cni
fierce-summer-6167
06/29/2022, 3:08 PMcurl -sfL <https://get.rke2.io> | INSTALL_RKE2_VERSION=1.22.9+rke2r2 sh -
but the installation script reports "No package rke-server-1.22.9-rke2r2 available". This release exists:
https://github.com/rancher/rke2/releases/tag/v1.22.9%2Brke2r2
I've tried various combinations of INSTALL_RKE2_CHANNEL, INSTALL_RKE2_VERSION, and INSTALL_RKE2_COMMIT, with no change in behavior. Is this expected, or should I be able to install a specific release?ambitious-plastic-3551
06/29/2022, 4:04 PM