Hello! I am running Rancher 2.6.6 and I'm having trouble running

kubectl exec

using the Rancher generated Kubeconfig. The command results in a blank error message:

Error from server:

. Upon further investigation, it seems that the

exec

command is unable to post to the Rancher proxy for the API server, and it receives a

403 forbidden

response:

POST https://<rancher-url>/k8s/clusters/local/api/v1/namespaces/<namespace>/pods/<pod>/exec?command=sh&container=<container>&stdin=true&stdout=true&tty=true 403 Forbidden in 238 milliseconds

All of our requests are routed through nginx, and we've confirmed that the nginx ingress receives the same 403 error. However, turning on trace logs for the Rancher pods, it appears that the message was never processed in the pod. We've also verified that our RBAC rules are sufficient for executing an

exec

command as all

kubectl auth can-i

prompts return "yes." Please let me know if you have any ideas on troubleshooting this issue. Thank you!

For a Rancher API request like that, you would need to use a bearer token - do you have a token? https://rancher.com/docs/rancher/v2.6/en/user-settings/api-keys/#creating-an-api-key

Yes. The Rancher-generated kubeconfig has a token that's used as a bearer.

kubectl exec

has two API calls through Rancher. The first request is a

GET

request to list all the containers in a pod, and the second is a

POST

request to run a command on the found container. The

GET

request returns a

200

status, while the

POST

request returns a

403

. This means the token is working for some API requests.

Does the same command work if you do it from cluster explorer? from the kubectl shell in the top nav?

Yes the same

kubectl exec

command works from the Rancher shell in the cluster explorer. I’m only running into this issue on my local machine using the generated kubeconfig for that same cluster.