This message was deleted.

log into the node and check the systemd logs for rancher-system-agent and rke2-server

So I docker system prune -f --volumes -a && rm -rf /var/lib/rancher then restart node remove the cluster. recreate one by create --> custom --> basics: Kubernetes Version v1.27.8+rke2r1, I don't change anything just for test the basic setup. I deploy one one node

[root@server1 ~]# curl --insecure -fL <https://192.168.137.50:8443/system-agent-install.sh> | sudo sh -s - --server <https://192.168.137.50:8443> --label '<http://cattle.io/os=linux|cattle.io/os=linux>' --token clhzjsbz97bf86r48r7b5w78c265f6cdjdv7jts6pwm8z8hlr29ztq --ca-checksum 2ec3f6509b0ebcba9cce27a4711b8c12d6fcda0477c7194488393d0b37f15557 --etcd --controlplane --worker
 % Total  % Received % Xferd Average Speed  Time  Time   Time Current
                 Dload Upload  Total  Spent  Left Speed
100 30863  0 30863  0   0 1883k   0 --:--:-- --:--:-- --:--:-- 1883k
[INFO] Label: <http://cattle.io/os=linux|cattle.io/os=linux>
[INFO] Role requested: etcd
[INFO] Role requested: controlplane
[INFO] Role requested: worker
[INFO] Using default agent configuration directory /etc/rancher/agent
[INFO] Using default agent var directory /var/lib/rancher/agent
[INFO] Determined CA is necessary to connect to Rancher
[INFO] Successfully downloaded CA certificate
[INFO] Value from <https://192.168.137.50:8443/cacerts> is an x509 certificate
[INFO] Successfully tested Rancher connection
[INFO] Downloading rancher-system-agent binary from <https://192.168.137.50:8443/assets/rancher-system-agent-amd64>
[INFO] Successfully downloaded the rancher-system-agent binary.
[INFO] Downloading rancher-system-agent-uninstall.sh script from <https://192.168.137.50:8443/assets/system-agent-uninstall.sh>
[INFO] Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO] Generating Cattle ID
[INFO] Cattle ID was already detected as e6707cee5c32618230e0d88075efafcb8652b1733bb90dd013e340f020a6ce5. Not generating a new one.
[INFO] Successfully downloaded Rancher connection information
[INFO] systemd: Creating service file
[INFO] Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO] Enabling rancher-system-agent.service
[INFO] Starting/restarting rancher-system-agent.service
[root@server1 ~]#

The logs

May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=warning msg="Failed to get image from endpoint: Get \"<https://index.docker.io/v2/>\": dial>
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=fatal msg="failed to get runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r|index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r>>
May 07 14:55:04 server1 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
May 07 14:55:04 server1 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
May 07 14:55:04 server1 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
May 07 14:55:09 server1 systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
May 07 14:55:09 server1 systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 1500.
May 07 14:55:09 server1 systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
May 07 14:55:09 server1 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
May 07 14:55:09 server1 sh[128522]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
May 07 14:55:09 server1 sh[128523]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
May 07 14:55:09 server1 rke2[128529]: time="2024-05-07T14:55:09+02:00" level=warning msg="not running in CIS mode"
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=warning msg="Failed to get image from endpoint: Get \"<https://index.docker.io/v2/>\": dial>
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=fatal msg="failed to get runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r|index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r>>
May 07 14:55:11 server1 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
May 07 14:55:11 server1 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
May 07 14:55:11 server1 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
...skipping...
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=warning msg="Failed to get image from endpoint: Get \"<https://index.docker.io/v2/>\": dial>
May 07 14:55:04 server1 rke2[128447]: time="2024-05-07T14:55:04+02:00" level=fatal msg="failed to get runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r|index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r>>
May 07 14:55:04 server1 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
May 07 14:55:04 server1 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
May 07 14:55:04 server1 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
May 07 14:55:09 server1 systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
May 07 14:55:09 server1 systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 1500.
May 07 14:55:09 server1 systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
May 07 14:55:09 server1 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
May 07 14:55:09 server1 sh[128522]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
May 07 14:55:09 server1 sh[128523]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
May 07 14:55:09 server1 rke2[128529]: time="2024-05-07T14:55:09+02:00" level=warning msg="not running in CIS mode"
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=warning msg="Failed to load runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rk|index.docker.io/rancher/rke2-runtime:v1.27.8-rk>>
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=warning msg="Failed to get image from endpoint: Get \"<https://index.docker.io/v2/>\": dial>
May 07 14:55:11 server1 rke2[128529]: time="2024-05-07T14:55:11+02:00" level=fatal msg="failed to get runtime image <http://index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r|index.docker.io/rancher/rke2-runtime:v1.27.8-rke2r>>
May 07 14:55:11 server1 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
May 07 14:55:11 server1 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
May 07 14:55:11 server1 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
lines 3923-3982/3982 (END)

yeah you’ll need to set up a private registry mirror that has all the images on it, and configure that in the registry section when provisioning the cluster.

I use https://docs.rke2.io/install/airgap#tarball-method for setting-up RKE2.