This message was deleted.

did you check the output from systemd process? You might not have the images pre-loaded for containerd to import.

I have defined a registries.yaml which should rewrite the images correctly. In the docs it says preload the archives into the images folder OR provide a registries config

I have not used the registry myself, so I cannot comment how it actually downloads the images. Can you try to put the images in the folder to figure out if that is what is missing? If that works, you can investigate the registry - perhaps it cannot download the images for some reason

https://gist.github.com/tboerger/2522f21ea61acc4617365905d144dc49 I also don't get the errors entirely... I think it should start containerd and etcd and so on, but it just prints errors like etcd isn't starting and the server doesn't get read with messages like

Waiting to retrieve agent configuration; server is not ready: yaml: line 11: did not find expected key

I assume this is a fresh installed cluster

Yes it's fresh installed. I can't easily get the images now, I need to talk to the engineer to add it to the mirror since I can't download the archives on my own.

In my experience, it is normal that you get lots of errors at startup. Things start and keep failing because of unmet dependencies until everything is up and running.

I can't find any containerd process and there are also no binaries (crictl, kubectl etc) created so far.

For example with the following command:

# /var/lib/rancher/rke2/bin/crictl images

It should, but it doesn'T https://gist.github.com/tboerger/2b4bcf5e2541cc02acb6f54d4c0862a9

in my case bin is a symlink to /var/lib/rancher/rke2/data/v1.29.3-rke2r1-106ea6e3639d/bin\

You are talking about https://get.rke2.io? This doesn't install kubectl etc

It does install everything needed, including kubectl

The

curl -sfL <https://get.rke2.io> | sh -

script just installs rke2, rke2-killall.sh and rke2-uninstall.sh, nothing else

There are other files in /usr/local as well

Yeah but looks like it's not doing it for me on the airgapped environment 😞

It needs the images to proceed, so you have to preinstall them or have a local repository which can download them from the internet

I have a local registry where I can get them

I have installed it in one air-gapped environment so far, so not much experience, but it worked

I have never installed Kubernetes on air-gapped environments before

I would suggest trying it on a local vm (in your pc) with linux without network connection. When that works, you can figure out why it does not work in the more restricted environment

I guess the images tarball doesn't matter as long as there is not any containerd process running at all?

What do you see with

systemctl status rke2-server.service

?

# systemctl status rke2-server.service
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/etc/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
     Active: activating (start) since Mon 2024-04-29 10:19:03 UTC; 39s ago
       Docs: <https://github.com/rancher/rke2#readme>
    Process: 80027 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
   Main PID: 80029 (rke2)
      Tasks: 10
     Memory: 115.2M
        CPU: 45.389s
     CGroup: /system.slice/rke2-server.service
             └─80029 "/usr/local/bin/rke2 server"

+ logs

I see lots of containerd-shim-runc-v2 (the PODs), but also containerd, kublet

I see mostly the errors related to not running etcd

Ok, this looks like only the rke2 process is started, and it did not start containerd

The initial master doesn't get a server url?

something like "server: https://fqdn:9345"

At least that's what I have read

OK, maybe I am wrong

Which private CA? 😄

A sorry, it is the internal ca from kubernetes which is causing this error. This is automatically generated.

Just tried to add server and token, than it tries to fetch the certs from the server url. So it really have to be undefined for the initial master. Everything on ubuntu 22.04.