This message was deleted.
# rke2a
adamant-kite-43734
11/26/2023, 8:37 PMThis message was deleted.
c
creamy-pencil-82913
11/26/2023, 8:54 PMThat is a really old version of rke2... I'm confused though, you said the servers all started up fine. The error message you're showing here would not happen on an agent (worker) node. Did you go back and change something on the servers after updating the server address on the agents?
s
some-book-73639
11/26/2023, 8:54 PMThe master nodes started fine.
some-book-73639
11/26/2023, 8:54 PMThe worker nodes didn't
some-book-73639
11/26/2023, 8:55 PMWhen checking the 50-rancher.yaml file, the server referenced there doesn't exist, so I changed it to one of the working master nodes, then started rke and that's the error that comes out
some-book-73639
11/26/2023, 8:56 PMThis cluster was brought up about a year ago, and since it's used for observability, we don't have maintenance windows on it often.
some-book-73639
11/26/2023, 8:58 PMshit - I think my server node IP's may have changed.
some-book-73639
11/26/2023, 8:59 PMthat subnet has dhcp on it and someone didn't static reserve the IP's for those machines
c
creamy-pencil-82913
11/26/2023, 8:59 PMYeah that would definitely break things. Server IPs need to be static, or DHCP reservations
creamy-pencil-82913
11/26/2023, 9:00 PMThe error message is from the server code, youre sure you're not trying to start the server service on a worker node?
s
some-book-73639
11/26/2023, 9:00 PMNow I guess I have to find the old IP addresses of all the nodes. Funny the master nodes came up
c
creamy-pencil-82913
11/26/2023, 9:01 PMAn agent (worker) will not say "preparing server" in its logs
s
some-book-73639
11/26/2023, 9:01 PMI've gotten about 6 hours of sleep in the last 4 days. What can I do to verify the worker node
c
creamy-pencil-82913
11/26/2023, 9:02 PMJust confirm that you're starting rke2-agent on the workers, not rke2-server? If they are not supposed to be servers...
s
some-book-73639
11/26/2023, 9:02 PMrke-agent not found
c
creamy-pencil-82913
11/26/2023, 9:03 PMWhat are you running exactly?
creamy-pencil-82913
11/26/2023, 9:03 PMAnd you said these are agent/worker nodes not servers, are you sure of that?
s
some-book-73639
11/26/2023, 9:03 PMIt's a Rancher RKE2 cluster
some-book-73639
11/26/2023, 9:06 PM483042
some-book-73639
11/26/2023, 10:48 PMSomething is completely messed up. Set the nodes back to the proper IP, rebooted and Rancher is just not updating the status or anything.
c
creamy-pencil-82913
11/27/2023, 12:10 AMAre all the servers running? Can you run
kubectl get nodecreamy-pencil-82913
11/27/2023, 12:11 AMIt might be helpful to note which ones are servers and which ones are agents
s
some-book-73639
11/27/2023, 1:07 AMcluster is ready in kubectl
some-book-73639
11/27/2023, 1:14 AMand now it's showing up well in rancher.
some-book-73639
11/27/2023, 1:14 AMRancher is this mysterious thing that may look broken but eventually fixes itself.
c
creamy-pencil-82913
11/27/2023, 1:18 AMyeah the cattle cluster agent needs to be running in the downstream cluster for things to sync up in Rancher
s
some-book-73639
11/27/2023, 1:18 AMThanks for listening to me complain and live stream troubleshoot. Glad it's up.
2 Views