https://rancher.com/ logo
Join Slack
Powered by
Hello, after rotating the credentials of my GKE, R...
# general
r
Hello, after rotating the credentials of my GKE, Rancher no longer sees the imported cluster as healthy it shows "Cluster agent is not connected". I cannot see anything helpful in the logs, official docs or github issues. Is there a way to force refresh the cluster import since the GKE has new credentials ? I have tried to edit config and save to force a reconciliation without success. Thanks πŸ™
I know that the credentials rotation would have broken the creds for Rancher but unsure how to update them πŸ˜•
b
The cloud credentials?
r
I don't have any since they are imported clusters πŸ™‚
Any other ideas ? 😒
b
I don't think I've ever imported a GKE cluster so I have no idea. I thought you had to be running a typical VM for registration/adoption, which would be GCP. I've only ever had cloud credentials for GKE. Β―\_(ツ)_/Β―
r
Yeah
b
Do you have docs you followed to do it?
r
I am just surprised there's no docs for how to re-register after rotating credentials
I do not want to delete and re-import as I'd have to reconcile all my terraform states πŸ˜•
Exploring the cluster works fine in Rancher though. It's just on the home page it's not available.
b
If you can explore the cluster then it's working? Maybe it's just the rancher-agent that needs to be restarted?
r
I have bounced the agent pods several times
b
Check the logs
r
I think explore works because it's existing and just queries the kube api directly
ya nothing spicy in the logs 🀷
b
The logs show it's calling home and connected?
r
Just infinite
Copy code
[ERROR] failed to start cluster controllers c-j7m6t: context canceled
[INFO] Stopping cluster agent for c-j7m6t
and the odd
Copy code
[ERROR] error syncing 'c-j7m6t': handler cluster-deploy: cannot connect to the cluster's Kubernetes API, requeuing
It's because the creds were rotated on the cluster and I have no clue how to update them in Rancher πŸ™‚
b
The agent should be from inside your GKE cluster
r
I could delete the secrets and see what happens ?
b
I would poke around GKE and find the agent logs there
You should be able to fix it from that side. (I think)
r
I am in those logs, they are up and running and don't report errors
b
It seems like Rancher's creds are working since you can explore the cluster.
r
I need to find docs that talks about the creds components for agents
yes it's confusing right
image.png
b
cluster agent is not connected
r
image.png
b
The issue seems to be from GKE -> Rancher.
r
network is fine, the agents logs shows:
Copy code
ERROR 2025-10-29T16:27:24.723912946Z [resource.labels.containerName: cluster-register] INFO: <https://rancher.DOMAIN.COM/ping> is accessible
ERROR 2025-10-29T16:27:24.739431320Z [resource.labels.containerName: cluster-register] INFO: <http://rancher.limep.net|rancher.limep.net> resolves to 10.0.2.21
b
Seems like a bad config
r
I fixed it by running
Copy code
k delete secret cattle-token  -n cattle-system
k delete secret cattle-token-r9fn2   -n cattle-system
In case anyone else has the same issue.
Now the agents are back up healthy βœ…
However I am unable to execute shell or view logs on workloads πŸ˜‚
Ended up having to delete and reimport cluster 😞
https://github.com/rancher/rancher/issues/52555 hopefully someone from Rancher can tell me what I'm doing wrong πŸ™
81 Views
Open in Slack
PreviousNext
Powered by