Hi, I’ve started skilling up on k8s a few weeks back. I landed at a client with high security standards...where I am apparently not allowed to share log files with the outside world :( I need to deploy three k8s clusters on bare metal RedHat 7 servers, with rke2 (v1.26.2+rke2r1), in air gap mode. I am facing two issues: A - The rke2 install does not complete / I don’t get the prompt back. But I do get: 1) static pods installed, in Running or Completed status 2) a master node up on which I can run a test pod B - Usually after several hours (trigger is not clear yet), the status of the master node will go from Ready to NotReady. When I describe the node then, I consistently get this error “NodeStatusUnknown Kubelet stopped posting node status.” Expected behavior: A - the rke2 install process would complete, such that I can enable & start the rke2-server service B - the master node remains Ready The client is getting impatient...I hope you guys can help me unlock the situation, despite their restriction on sharing complete log files...

are you the person that opened an issue and then edited the post to delete all the info?

that was me brandond...sorry!

Unfortunately we can’t really help without logs. You can try looking at the logs I requested yourself to try to see if there’s anything in there that would suggest what’s going on.

I can share key parts of the logs, but can’t share complete log files 😞

those aren’t errors though, that’s just a debug message about an internal connection

you’re right, these are just debug messages

if the kubelet isn’t posting ready status there should be something in the kubelet log just before that happens, there’s nothing in there about it?

My understanding is that the master node goes to NotReady status if the rke2-server install process dies, or if I kill it...

well yes, if you stop the service then the kubelet stops

the thing is I’ve never reached a point where the rke2-server service can be enabled / started....its supporting systemd file is not created

why not? is the installer erroring out?

no, the installer just stays busy for hours on end spitting the kind of messages I posted above

Are you confusing rke2 for the installer?

Yes, in the airgap install page, we went with the RKE2 binary install method

ok, so you haven’t used the installer at all

not the install.sh method, no

yes, technically you have installed RKE2 by downloading the binary and running it, but if you don’t use the install script then you need to create the systemd unit yourself.

OK, I understand better brandon, thanks! We were told by another team they had used the binary method, but isn’t the install.sh the preferred install method?

you can use the binary method just fine, you just have to figure out how to get it run as a service so that it doesn’t exit when you log out

OK, but I’m scripting this entire process with a shell script...how could I tell my shell script it’s OK to kill the rke2 server process and install the systemd service?

I wouldn’t have your shell script run

rke2 server

at all. Just copy the systemd unit file onto the host and then use systemctl to start the service.

I think I see what you mean...what you describe above “Just copy the systemd unit file onto the host and then use systemctl to start the service.” seems way easier, so I’m surprised this is not described here https://docs.rke2.io/install/airgap as the best / recommended option. I’m a bit puzzled...

Honestly it’s not something that I’ve had to clarify before, I’m not sure if most folks figure it out on their own or what.

I’ll look into this later tonight or tomorrow...it’s been a long day here...like the previous days...

np, gl

A support contract w rancher could include an nda which should allow you to share logs 😉

I do sometimes find myself questioning my choices when I'm helping someone deploy one of our products, who's clearly being paid to do so despite not understanding the product, and does not work for us or fund development of the product.