https://rancher.com/ logo
Title
p

polite-translator-35958

05/25/2023, 3:09 PM
I’m trying to do a fresh rke2 install on RHEL8.8 (the same way I did it on RHEL8.7) and it seems like something has changed.
systemctl start rke2-server
eventually times out and fails. It seems like it’s waiting for etcd startup which never happens. I’m guessing something changed in the rhel release, but I’m failing to grok what the issue is. Anyone seen this?
h

hundreds-evening-84071

05/25/2023, 4:26 PM
which RKE2 version? looks like RHEL 8.8 isn't on support matrix yet... https://www.suse.com/suse-rke2/support-matrix/all-supported-versions/rke2-v1-26/
p

polite-translator-35958

05/25/2023, 4:54 PM
RKE2 v1.25.9~rke2r1
But yeah, I see what you mean….not listed…
g

gray-lawyer-73831

05/25/2023, 6:32 PM
Yeah this hasn’t been validated as working yet, but I don’t see any reason why it shouldn’t. Couple of things to check: https://docs.rke2.io/known_issues#networkmanager Also check container-selinux version as there have been multiple issues around this recently with a version bump there that caused rke2-selinux to fail:
rpm -qi container-selinux
p

polite-translator-35958

05/25/2023, 7:08 PM
Yeah, for some reason I didn’t even consider that a minor update would not work. 🤦 Confirming that
container-selinux
went from
v2.189.0-1
on a working RHEL8.7 system to
v2.205.0-2
on non-working RHEL8.8 system.
And also confirming that I do have the
/etc/NetworkManager/conf.d/rke2-canal.conf
file as described on that known issues page.
g

gray-lawyer-73831

05/25/2023, 7:11 PM
yeah container-selinux is the issue! If you install v2.189 then it should all work fine! We’re in the process of making fixes for that on our side, but for now you should be able to just force it before installing rke2 with something like:
yum install container-selinux-2:2.189.0-1.module+el8.7.0+17824+66a0202b
p

polite-translator-35958

05/25/2023, 7:11 PM
ahh. okay. I will give that a whirl. Thanks so much!
🤞 1
Yep! That seems to have done the trick. Thanks again!
🥂 2
b

billowy-afternoon-45803

05/25/2023, 9:20 PM
I am running into issue deploying also though I am only trying to deploy to a simple virtualbox vm. I removed the package described above and downgraded it [2.189.0](https://centos.pkgs.org/8-stream/centos-appstream-aarch64/container-selinux-2.189.0-1.module_el8.7.0+1217+ea57d1f1.noarch.rpm.html) and tried again after rebooting, still no luck. I next attempted simply disabling selinux and rebooting and still no dice. I can provide logs if anyone is interested, or is willing to help, it would be much appreciated.
h

hundreds-evening-84071

05/25/2023, 9:26 PM
have you looked at
journalctl -u rke2-server
to see it contains any pointers?
b

billowy-afternoon-45803

05/25/2023, 9:44 PM
Mostly the following:
T15:30:14-06:00" level=info msg="Waiting for API server to become available"
T15:30:14-06:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
T15:30:16-06:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://192.168.56.3:9345/v1-rke2/readyz>: 500 Internal Server Error"
T15:30:19-06:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
T15:30:21-06:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://192.168.56.3:9345/v1-rke2/readyz>: 500 Internal Server Error"
T15:30:24-06:00" level=info msg="Container for etcd not found (no matching container found), retrying"
T15:30:24-06:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
T15:30:26-06:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://192.168.56.3:9345/v1-rke2/readyz>: 500 Internal Server Error"
"ts":"2023-05-25T15:30:29.464-0600","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc00079dc00/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = Deadline>
"ts":"2023-05-25T15:30:29.465-0600","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
T15:30:29-06:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
T15:30:31-06:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://192.168.56.3:9345/v1-rke2/readyz>: 500 Internal Server Error"
T15:30:34-06:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Nothing very specific, as far as I can telll RKE2 isnt even starting any containers checking the /var/log/containers log directory yields an empty folder.
Sorry thought the log would be collapsed
p

polite-translator-35958

05/26/2023, 4:28 AM
Those log entries look like what I was seeing. I will add that when I just downgraded the container-selinux package, it didn’t seem to solve the issue. Was testing with an ec2 instance, so I just whacked it an made a new one and installed the lower version from the start. Then it worked as I expected….
h

hundreds-evening-84071

05/26/2023, 1:16 PM
So perhaps... Try and run: rke2-uninstall Make sure these directories are cleaned up after uninstall (they should but should confirm) /etc/rancher/ /var/lib/rancher /usr/local/lib/systemd/system/ ( make sure rke2-server service scripts are gone) reboot if container-selinux got removed then reinstall then reinstall rke2 and see if this works?