I've lost access to Rancher in one of our managed clusters. I have the Rancher-generated kubeconfig file (that relies on the rancher service) so that's of no use to me to get access to the k8s cluster I was controlling. I have access to all the cluster nodes but the config does not give me direct access to the cluster it was managing. If we assume I cannot restore Rancher, is it possible get out of this mess and manufacture a kubeconfig file that will let me can access the managed cluster directly? One that bypasses the original Rancher service?

What kind of cluster does your Rancher instance reside on?

The cluster's on on on-prem OpenStack cluster - I have access to the VMs' so can see the docker containers running on those nodes. Sadly the "authorised endpoint" feature wasn't set, so I don't have a config with the control endpoint. But I see the

kube-apiservice

and its container ENV. SO I just wondered whether the certs or other material is available to me to manufacture a downstream config by hand?

Is it an RKE2/K3s cluster?

It's an rke v1 k8s cluster

When you initially ran

rke up

it would have generated a kubeconfig that auths directly to the API server. Failing that, ssh to one of your rancher management cluster nodes. Install kubectl and run:

kubectl --kubeconfig $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl/kubecfg-kube-node.yaml get configmap -n kube-system full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r .currentState.certificatesBundle.\"kube-admin\".config | sed -e "/^[[:space:]]*server:/ s_:.*_: \"<https://127.0.0.1:6443>\"_" > kubeconfig_admin.yaml
kubectl --kubeconfig kubeconfig_admin.yaml get nodes

Wonderful! That's worked! Thank you.

You're welcome.