Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Title
b
blue-needle-61113
11/17/2022, 7:34 PMHi Lads, was someone able to succesfuly deploy k8s cluster using rancher 2.6.9 , rke2 and vsphere. I’m nearly there but ui displays that it stucked on “Updating”. “Non-ready bootstrap machine(s) test-node-selectors-pool1-644585c7ff-qgrtx and join url to be available on bootstrap node” Thanks for any help
d
damp-painting-69352
11/17/2022, 8:33 PMYes i've got this working, what are you stuck on exactly
b
blue-needle-61113
11/18/2022, 8:49 AMHi Zach, Let’s assume I want to deploy basic cluster configuration. 3 pools - 1 for control plane, 1 for etcd and 1 worker. Not for production just for test. My problem is that only control plane and etcd nodes are up and connected to rancher. Worker node has rancher-system-agent but log is stopped at this:
root@test-node-selectors-pool1-6fd7de46-mbhng:/home/ubuntu# journalctl -u rancher-system-agent
-- Logs begin at Thu 2022-11-17 12:14:12 UTC, end at Thu 2022-11-17 20:19:42 UTC. --
Nov 17 12:15:47 test-node-selectors-pool1-6fd7de46-mbhng systemd[1]: Started Rancher System Agent.
Nov 17 12:15:47 test-node-selectors-pool1-6fd7de46-mbhng rancher-system-agent[1155]: time="2022-11-17T12:15:47Z" level=info msg="Rancher System Agent version v0.2.13 (4fa9427) is startin>
Nov 17 12:15:47 test-node-selectors-pool1-6fd7de46-mbhng rancher-system-agent[1155]: time="2022-11-17T12:15:47Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Nov 17 12:15:47 test-node-selectors-pool1-6fd7de46-mbhng rancher-system-agent[1155]: time="2022-11-17T12:15:47Z" level=info msg="Starting remote watch of plans"
Nov 17 12:15:47 test-node-selectors-pool1-6fd7de46-mbhng rancher-system-agent[1155]: time="2022-11-17T12:15:47Z" level=info msg="Starting /v1, Kind=Secret controller"When I deploy node with all roles in every node, I can explore the cluster but it is stucked in “Updated” status, never comming to Active.
d
damp-painting-69352
11/21/2022, 2:37 PMEnsure firewall ports are configured between nodes on your template. Also grep the logs for "image" to check for any image pull failures
b
blue-needle-61113
11/23/2022, 9:29 PMHi Zach, I totally removed firewall for testing purposes. I checked the logs and nothing about images. I can see that there is a problem with Worker role in general. For example if I setup node as a Worker it has only rancher-system-agent sentinel process up and running in relation to rke2. As far as I can see it should pull images and run agent - nothing like that happens
d
damp-painting-69352
11/23/2022, 9:55 PMthe rancher-system-agent process does need to pull the installation image for rke2 from a remote repo
that is a pull always command 😞
rancher/system-agent-installer-rke2:v1.23.10-rke2r1 this is example image it is trying to pull
you can check the logs for the server
journalctl -flu rancher-system-agenti think the rancher-system-agent logs only shows the pull failure once, so it pretty hard to find
b
blue-needle-61113
11/23/2022, 10:00 PMhi Zach
thats my entire log from the agent
root@test2211-3-worker-299cecfb-mf4m4:/home/ubuntu# journalctl -flu rancher-system-agent
-- Logs begin at Tue 2022-11-22 11:15:11 UTC. --
Nov 22 11:15:20 test2211-3-worker-299cecfb-mf4m4 systemd[1]: Started Rancher System Agent.
Nov 22 11:15:20 test2211-3-worker-299cecfb-mf4m4 rancher-system-agent[1234]: time="2022-11-22T11:15:20Z" level=info msg="Rancher System Agent version v0.2.13 (4fa9427) is starting"
Nov 22 11:15:20 test2211-3-worker-299cecfb-mf4m4 rancher-system-agent[1234]: time="2022-11-22T11:15:20Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Nov 22 11:15:20 test2211-3-worker-299cecfb-mf4m4 rancher-system-agent[1234]: time="2022-11-22T11:15:20Z" level=info msg="Starting remote watch of plans"
Nov 22 11:15:21 test2211-3-worker-299cecfb-mf4m4 rancher-system-agent[1234]: time="2022-11-22T11:15:21Z" level=info msg="Starting /v1, Kind=Secret controller"d
damp-painting-69352
11/23/2022, 10:02 PMif you check in /var/lib/rancher/rke2/ are there folders
b
blue-needle-61113
11/23/2022, 10:02 PMonly on not Workers node
on the worker role node I’ve got :
d
damp-painting-69352
11/23/2022, 10:04 PMokay that agent folder is the rancher-system-agent
so since you don't see the other folders rke2 hasn't installed and the container hasn't pulled
b
blue-needle-61113
11/23/2022, 10:05 PMdo you have any idea why Workers are the only affected
?
I tried a few versions of rancher
d
damp-painting-69352
11/23/2022, 10:05 PMdepending on how you are trying to provision them... yes there could be many reason
it's probably not the version of rancher that is the issue
if you have gotten this far i would check cloud-init logs next
to make sure it is getting the user-data from the iso
next is the node-pool. it could be an improper setting somewhere in teh cloud-init for that patricular node-pool
anyway, catch me next week if you still haven't figured it out, Happy thanksgiving
b
blue-needle-61113
11/23/2022, 10:10 PMahh ok, Thank you very much for that help, enjoy your time, I’m from Poland so I forgot about Thanksgiving
d
damp-painting-69352
11/23/2022, 10:10 PMnp, yep silly American holiday. Hope you find that needle in the haystack
b
blue-needle-61113
11/23/2022, 10:11 PMI dont think its silly, tradition is tradition, good time with family, friends, and head out of computers 🙂 Enjoy and thanks again
❤️ 1
s
square-policeman-85866
12/02/2022, 5:13 PMHi, I’m gettin the exact same issue building an rke2 cluster on vsphere. Where you able to figure out the issue?
d
damp-painting-69352
12/05/2022, 3:59 PM@square-policeman-85866 Can you provide logs, or screenshots of the issue
s
square-policeman-85866
12/13/2022, 4:21 PMHi Zach. Is there a specific log I can look into ?
d
damp-painting-69352
12/13/2022, 4:22 PMYou should create a new thread and include logs of the issue you are seeing. No idea what this thread is about at this point
s
square-policeman-85866
12/13/2022, 4:23 PMWill do thanks