https://rancher.com/ logo
#rke2
Title
# rke2
e

early-sugar-56788

03/22/2023, 6:08 PM
Hi, I’ve started skilling up on k8s a few weeks back. I landed at a client with high security standards...where I am apparently not allowed to share log files with the outside world :( I need to deploy three k8s clusters on bare metal RedHat 7 servers, with rke2 (v1.26.2+rke2r1), in air gap mode. I am facing two issues: A - The rke2 install does not complete / I don’t get the prompt back. But I do get: 1) static pods installed, in Running or Completed status 2) a master node up on which I can run a test pod B - Usually after several hours (trigger is not clear yet), the status of the master node will go from Ready to NotReady. When I describe the node then, I consistently get this error “NodeStatusUnknown Kubelet stopped posting node status.” Expected behavior: A - the rke2 install process would complete, such that I can enable & start the rke2-server service B - the master node remains Ready The client is getting impatient...I hope you guys can help me unlock the situation, despite their restriction on sharing complete log files...
c

creamy-pencil-82913

03/22/2023, 6:09 PM
are you the person that opened an issue and then edited the post to delete all the info?
e

early-sugar-56788

03/22/2023, 6:10 PM
that was me brandond...sorry!
c

creamy-pencil-82913

03/22/2023, 6:10 PM
Unfortunately we can’t really help without logs. You can try looking at the logs I requested yourself to try to see if there’s anything in there that would suggest what’s going on.
e

early-sugar-56788

03/22/2023, 6:11 PM
I can share key parts of the logs, but can’t share complete log files 😞
This makes it hard on you, and me, but I hope we can work it out
Trust me, I wish I/we didn’t have this problem...but this forum seems like my best option at this stage
The rke2 installer remains stuck, trying the following over and over:
time=“2023-03-20T205927+01:00” level=debug msg=“Wrote ping” time=“2023-03-20T205928+01:00” level=debug msg=“Tunnel server handing HTTP/1.1 CONNECT request for //10.42.0.12:10250 from 127.0.0.1:45414" time=“2023-03-20T205928+01:00” level=debug msg=“Tunnel server egress proxy dialing 10.42.0.12:10250 directly” time=“2023-03-20T205932+01:00” level=debug msg=“Wrote ping”
We have been looking at the log files you mentioned in your reply, but the piece above is the most telling
c

creamy-pencil-82913

03/22/2023, 6:24 PM
those aren’t errors though, that’s just a debug message about an internal connection
e

early-sugar-56788

03/22/2023, 6:25 PM
you’re right, these are just debug messages
c

creamy-pencil-82913

03/22/2023, 6:25 PM
if the kubelet isn’t posting ready status there should be something in the kubelet log just before that happens, there’s nothing in there about it?
e

early-sugar-56788

03/22/2023, 6:28 PM
My understanding is that the master node goes to NotReady status if the rke2-server install process dies, or if I kill it...
But I’m wondering what can prevent the installer to complete, and give me the prompt back...because the install seems pretty complete / functional...
c

creamy-pencil-82913

03/22/2023, 6:32 PM
well yes, if you stop the service then the kubelet stops
you’re not intentionally stopping the service are you?
e

early-sugar-56788

03/22/2023, 6:34 PM
the thing is I’ve never reached a point where the rke2-server service can be enabled / started....its supporting systemd file is not created
c

creamy-pencil-82913

03/22/2023, 6:34 PM
why not? is the installer erroring out?
e

early-sugar-56788

03/22/2023, 6:35 PM
no, the installer just stays busy for hours on end spitting the kind of messages I posted above
but it’s not erroring out
c

creamy-pencil-82913

03/22/2023, 6:36 PM
Are you confusing rke2 for the installer?
those are not messages from the installer, those are messages from the rke2 server itself
it sounds like you’ve downloaded the rke2 binary and run that, instead of running the installer.
e

early-sugar-56788

03/22/2023, 6:40 PM
Yes, in the airgap install page, we went with the RKE2 binary install method
c

creamy-pencil-82913

03/22/2023, 6:40 PM
ok, so you haven’t used the installer at all
e

early-sugar-56788

03/22/2023, 6:40 PM
not the install.sh method, no
From https://docs.rke2.io/install/airgap#install-rke2, I read “RKE2 can be installed either by running the binary directly or by using the install.sh script.”
c

creamy-pencil-82913

03/22/2023, 6:45 PM
yes, technically you have installed RKE2 by downloading the binary and running it, but if you don’t use the install script then you need to create the systemd unit yourself.
If you just run it directly from the shell then yes, everything will stop when you kill the
rke2 server
process that you started from the shell. You’re not running the installer or starting a background service, you’re running the full server process right there, and when you exit it, you exit rke2.
You can find the systemd units here if you want to use them: https://github.com/rancher/rke2/tree/master/bundle/lib/systemd/system
e

early-sugar-56788

03/22/2023, 6:48 PM
OK, I understand better brandon, thanks! We were told by another team they had used the binary method, but isn’t the install.sh the preferred install method?
c

creamy-pencil-82913

03/22/2023, 6:48 PM
you can use the binary method just fine, you just have to figure out how to get it run as a service so that it doesn’t exit when you log out
e

early-sugar-56788

03/22/2023, 6:51 PM
OK, but I’m scripting this entire process with a shell script...how could I tell my shell script it’s OK to kill the rke2 server process and install the systemd service?
c

creamy-pencil-82913

03/22/2023, 6:51 PM
I wouldn’t have your shell script run
rke2 server
at all. Just copy the systemd unit file onto the host and then use systemctl to start the service.
rke2 server
isn’t an installer. It is the actual whole server itself. You are “installing” rke2 when you place the binary on the host.
e

early-sugar-56788

03/22/2023, 6:57 PM
I think I see what you mean...what you describe above “Just copy the systemd unit file onto the host and then use systemctl to start the service.” seems way easier, so I’m surprised this is not described here https://docs.rke2.io/install/airgap as the best / recommended option. I’m a bit puzzled...
c

creamy-pencil-82913

03/22/2023, 7:01 PM
Honestly it’s not something that I’ve had to clarify before, I’m not sure if most folks figure it out on their own or what.
e

early-sugar-56788

03/22/2023, 7:04 PM
I’ll look into this later tonight or tomorrow...it’s been a long day here...like the previous days...
I really appreciate your help brandon
c

creamy-pencil-82913

03/22/2023, 7:05 PM
np, gl
f

flaky-winter-94949

03/23/2023, 1:37 AM
A support contract w rancher could include an nda which should allow you to share logs 😉
💯 1
c

creamy-pencil-82913

03/23/2023, 1:50 AM
I do sometimes find myself questioning my choices when I'm helping someone deploy one of our products, who's clearly being paid to do so despite not understanding the product, and does not work for us or fund development of the product.
But that's open source I guess.
18 Views