https://rancher.com/ logo
Powered by
#general
Title
# general
e

enough-memory-12110

02/06/2023, 11:39 AM
Now the Rancher itself is unreachable neither from the web UI nor from the kubectl, how do I fix the problem, can anyone help?
b

brainy-tomato-18651

02/06/2023, 11:41 AM
I assume you already tried the same way
Copy code
k scale deploy rancher -n cattle-system --replicas=1
a

agreeable-oil-87482

02/06/2023, 11:41 AM
What kind of cluster is Rancher installed on? You could log onto one of the nodes, run kubectl locally (I assume the kubeconfig you're current using is pointing to the rancher auth proxy) and then scale up the deployment
e

enough-memory-12110

02/06/2023, 11:45 AM
@agreeable-oil-87482 Cluster is setup using the cluster.yaml file and imported into the cluster...
a

agreeable-oil-87482

02/06/2023, 11:45 AM
so you ran 
rke up
?
e

enough-memory-12110

02/06/2023, 11:45 AM
It was not me, but yes...
a

agreeable-oil-87482

02/06/2023, 11:46 AM
Running that will generate a kubeconfig that will auth to the apiserver of that cluster directly. Use that to increase the deployment replica count
e

enough-memory-12110

02/06/2023, 11:46 AM
how do I do that exactly?
rke config
you mean?
a

agreeable-oil-87482

02/06/2023, 11:46 AM
Do you have the kubeconfig that was generated by 
rke up
?
e

enough-memory-12110

02/06/2023, 11:47 AM
Yes, I do see the kubeconfig file inside that directory...
a

agreeable-oil-87482

02/06/2023, 11:47 AM
point kubectl to that file - can you get a list of Pods?
e

enough-memory-12110

02/06/2023, 11:48 AM
Thank you so much, you saved my day... 8-)
❤️
a

agreeable-oil-87482

02/06/2023, 11:48 AM
No worries 👍
e

enough-memory-12110

02/06/2023, 11:48 AM
I ran the scale command but it didn't fix the problem, it needs some time maybe?
a

agreeable-oil-87482

02/06/2023, 11:49 AM
It'll take some time for the Rancher pod to spin up
k get po -n cattle-system
- what does it list?
e

enough-memory-12110

02/06/2023, 11:49 AM
Yeah, it needed some time, the web UI is back...
Thank you very much one more time... ❤️
a

agreeable-oil-87482

02/06/2023, 11:50 AM
Pods will have spun up but wouldn't have passed the readiness probe yet.
You're welcome
e

enough-memory-12110

02/06/2023, 11:50 AM
@agreeable-oil-87482 The reason why I ran the command is because I have a node in 
Removing
state...
Can you help with that too?..
a

agreeable-oil-87482

02/06/2023, 11:51 AM
In a downstream cluster?
e

enough-memory-12110

02/06/2023, 11:51 AM
Yes, the cluster is a 
custom
type cluster....
The rancher version is v2.6.2
And the node says "Removing, resources scheduled for deletion"...
But it has been more 4 hours now...
a

agreeable-oil-87482

02/06/2023, 11:53 AM
Which method did you use to delete the node? Ie from the rancher UI?
e

enough-memory-12110

02/06/2023, 11:53 AM
Yes, from the rancher UI...
When it got stuck I went to the kubectl and removed the node from there...
a

agreeable-oil-87482

02/06/2023, 11:53 AM
Is anything running on the node?
e

enough-memory-12110

02/06/2023, 11:53 AM
Now the cluster doesn't see the node but it's listed in the UI...
No, it was a clean system...
actually the cluster itself is a clean cluster, I just installed longhorn today and that's it...
But that node didn't join the longhorn installation...
a

agreeable-oil-87482

02/06/2023, 11:55 AM
How many nodes in the cluster and which role(s) does that node have? And in 
kubectl get no
it's not listed anymore?
e

enough-memory-12110

02/06/2023, 11:55 AM
yes, 
k get no
doesn't list the node, it was a regular worker node...
The cluster has 3 masters and 5 workers, this node was called 
kub-dev-worker01
...
a

agreeable-oil-87482

02/06/2023, 11:56 AM
Rancher > Cluster > Nodes > Edit YAML for the node that's supposed to be removed and delete any 
finalizers
e

enough-memory-12110

02/06/2023, 11:57 AM
I am getting 404 page not found when trying to save the file...
a

agreeable-oil-87482

02/06/2023, 11:58 AM
ctrl+f5 the page - is it still there?
e

enough-memory-12110

02/06/2023, 11:58 AM
I tried refreshing, it's still there, yes...
a

agreeable-oil-87482

02/06/2023, 12:00 PM
<https://<your-rancher-url>/dashboard/c/local/explorer/management.cattle.io.node> - is it listed here?
e

enough-memory-12110

02/06/2023, 12:01 PM
Yes, it's there...
a

agreeable-oil-87482

02/06/2023, 12:01 PM
Try deleting it from there using the three dots. If that fails, edit its yaml and delete the finalizers from there.
e

enough-memory-12110

02/06/2023, 12:02 PM
I tried deleting from there, no luck...
I tried editing the yaml but it's not an option...Only the View option is available...
a

agreeable-oil-87482

02/06/2023, 12:04 PM
Ah ok. In your rancher management cluster:
Copy code
kubectl get <http://node.management.cattle.io|node.management.cattle.io> -A
Then edit one, ie:
Copy code
kubectl edit <http://node.management.cattle.io|node.management.cattle.io> machine-fncw4 -n c-m-9q4wfk4z
And remove the finalizers
e

enough-memory-12110

02/06/2023, 12:06 PM
Awesome, it got disappeared now... 8-)
You are great, I can't explain how thankful I am...
❤️
a

agreeable-oil-87482

02/06/2023, 12:06 PM
No problem at all 👍
Powered by