Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
e
enough-memory-12110
02/06/2023, 11:39 AMNow the Rancher itself is unreachable neither from the web UI nor from the kubectl, how do I fix the problem, can anyone help?
b
brainy-tomato-18651
02/06/2023, 11:41 AMI assume you already tried the same way
k scale deploy rancher -n cattle-system --replicas=1a
agreeable-oil-87482
02/06/2023, 11:41 AMWhat kind of cluster is Rancher installed on? You could log onto one of the nodes, run kubectl locally (I assume the kubeconfig you're current using is pointing to the rancher auth proxy) and then scale up the deployment
e
enough-memory-12110
02/06/2023, 11:45 AM@agreeable-oil-87482 Cluster is setup using the cluster.yaml file and imported into the cluster...
a
agreeable-oil-87482
02/06/2023, 11:45 AMso you ran
rke upe
enough-memory-12110
02/06/2023, 11:45 AMIt was not me, but yes...
a
agreeable-oil-87482
02/06/2023, 11:46 AMRunning that will generate a kubeconfig that will auth to the apiserver of that cluster directly. Use that to increase the deployment replica count
e
enough-memory-12110
02/06/2023, 11:46 AMhow do I do that exactly?
rke configa
agreeable-oil-87482
02/06/2023, 11:46 AMDo you have the kubeconfig that was generated by
rke upe
enough-memory-12110
02/06/2023, 11:47 AMYes, I do see the kubeconfig file inside that directory...
a
agreeable-oil-87482
02/06/2023, 11:47 AMpoint kubectl to that file - can you get a list of Pods?
e
enough-memory-12110
02/06/2023, 11:48 AMThank you so much, you saved my day... 8-)
❤️
a
agreeable-oil-87482
02/06/2023, 11:48 AMNo worries 👍
e
enough-memory-12110
02/06/2023, 11:48 AMI ran the scale command but it didn't fix the problem, it needs some time maybe?
a
agreeable-oil-87482
02/06/2023, 11:49 AMIt'll take some time for the Rancher pod to spin up
k get po -n cattle-systeme
enough-memory-12110
02/06/2023, 11:49 AMYeah, it needed some time, the web UI is back...
Thank you very much one more time... ❤️
a
agreeable-oil-87482
02/06/2023, 11:50 AMPods will have spun up but wouldn't have passed the readiness probe yet.
You're welcome
e
enough-memory-12110
02/06/2023, 11:50 AM@agreeable-oil-87482 The reason why I ran the command is because I have a node in
RemovingCan you help with that too?..
a
agreeable-oil-87482
02/06/2023, 11:51 AMIn a downstream cluster?
e
enough-memory-12110
02/06/2023, 11:51 AMYes, the cluster is a
customThe rancher version is v2.6.2
And the node says "Removing, resources scheduled for deletion"...
But it has been more 4 hours now...
a
agreeable-oil-87482
02/06/2023, 11:53 AMWhich method did you use to delete the node? Ie from the rancher UI?
e
enough-memory-12110
02/06/2023, 11:53 AMYes, from the rancher UI...
When it got stuck I went to the kubectl and removed the node from there...
a
agreeable-oil-87482
02/06/2023, 11:53 AMIs anything running on the node?
e
enough-memory-12110
02/06/2023, 11:53 AMNow the cluster doesn't see the node but it's listed in the UI...
No, it was a clean system...
actually the cluster itself is a clean cluster, I just installed longhorn today and that's it...
But that node didn't join the longhorn installation...
a
agreeable-oil-87482
02/06/2023, 11:55 AMHow many nodes in the cluster and which role(s) does that node have?
And in
kubectl get noe
enough-memory-12110
02/06/2023, 11:55 AMyes,
k get noThe cluster has 3 masters and 5 workers, this node was called
kub-dev-worker01a
agreeable-oil-87482
02/06/2023, 11:56 AMRancher > Cluster > Nodes > Edit YAML for the node that's supposed to be removed and delete any
finalizerse
enough-memory-12110
02/06/2023, 11:57 AMI am getting 404 page not found when trying to save the file...
a
agreeable-oil-87482
02/06/2023, 11:58 AMctrl+f5 the page - is it still there?
e
enough-memory-12110
02/06/2023, 11:58 AMI tried refreshing, it's still there, yes...
a
agreeable-oil-87482
02/06/2023, 12:00 PM<https://<your-rancher-url>/dashboard/c/local/explorer/management.cattle.io.node> - is it listed here?
e
enough-memory-12110
02/06/2023, 12:01 PMYes, it's there...
a
agreeable-oil-87482
02/06/2023, 12:01 PMTry deleting it from there using the three dots. If that fails, edit its yaml and delete the finalizers from there.
e
enough-memory-12110
02/06/2023, 12:02 PMI tried deleting from there, no luck...
I tried editing the yaml but it's not an option...Only the View option is available...
a
agreeable-oil-87482
02/06/2023, 12:04 PMAh ok. In your rancher management cluster:
kubectl get <http://node.management.cattle.io|node.management.cattle.io> -Akubectl edit <http://node.management.cattle.io|node.management.cattle.io> machine-fncw4 -n c-m-9q4wfk4ze
enough-memory-12110
02/06/2023, 12:06 PMAwesome, it got disappeared now... 8-)
You are great, I can't explain how thankful I am...
❤️
a
agreeable-oil-87482
02/06/2023, 12:06 PMNo problem at all 👍