https://rancher.com/ logo
Join Slack
Powered by
When I execute this command to an rke2-based clust...
# general
b
When I execute this command to an rke2-based cluster running rancher, it consistently takes 60s to get a response...what's the best way to debug the issue?
Copy code
kubectl get --raw \
  '/apis/ext.cattle.io/v1/tokens?allowWatchBookmarks=true&resourceVersion=2903900033&watch=true' -v10
b
Where are you running it from?
your laptop > rancher > downstream proxy? On a cp node of the RKE cluster?
Laptop > RKE cluster directly?
rancher container?
b
I'm running from a laptop through ingress -> rancher
To be clear, it is specifically that resource that seems to be hitting some internal timeout before responding
b
If you ssh in to a node, then run it there, does it take the same amount of time?
b
As an example, this responds immediately every time
Copy code
kubectl get --raw \
  '/apis/ext.cattle.io/v1/kubeconfigs?allowWatchBookmarks=true&watch=true' -v10
b
That would at least tell us it's an issue with that cluster and nothing in the path to it.
👍 1
b
There may be other resources (the symptom of this is a completely unusable argocd connection to this cluster)
I've been digging into it for several days straight lol and this is my best clue so far after resolving several surface level issues
b
There might be extra calls for security reasons that are happening because kubeconfigs vs tokens and the airgap is causing the delay
I'm shooting from the hip into the dark here, but I still think running from inside the cluster is a good next step
👍 1
b
yeah, it's exactly 60s every time so it's hitting some internal timeout on something it seems
b
Might be checkin time from cluster reaching out to rancher
b
Yeah, lemme run from in cluster
bypassing rancher altogether (ie: using a service account token from the cluster directly) returns immediately
I happened to be in the rancher pod executing kubectl commands (I knew it would have full admin access)
b
I know nothing about this, but I fed it to Gemini and it thinks it has the answer: What is Happening Behind the Scenes When a user calls Rancher with a Rancher bearer token: 1. Auth Transcoding: Rancher authenticates your request, maps your Rancher user permissions to K8s RBAC rules, and opens a downstream request using Kubernetes User Impersonation headers (
Impersonate-User
, 
Impersonate-Group
). 2. Impersonation Timeout Handling: Rancher's proxy handler wraps watch connections in an internal HTTP context. 3. The Silent Failure: For 
<http://ext.cattle.io/v1/tokens|ext.cattle.io/v1/tokens>
, when Rancher impersonates your identity down to the Steve API layer, Steve checks RBAC to filter which tokens your user is allowed to watch. If the impersonated watch stream returns no events, Rancher's auth wrapper enforces a 60-second default request deadline/timeout on impersonated HTTP client contexts before abruptly dropping or flushing the connection. When you use a Service Account token directly: • No Impersonation Proxy: You present a direct cluster token (
system:serviceaccount:...
). • Rancher’s auth proxy does not wrap the connection in an impersonation context, so the HTTP stream stays open cleanly without the 60s forced flush/drop.
ai slop.txt
ai slop.txt
I couldn't reproduce it for rancher in my clusters, but I think that's probably because I have no token objects for some reason.
but TIL there's a Steve and a Norman in Rancher.
b
I've seen Norman but was unaware of Steve lol
I don't think that's the issue (none of those commands return any logs) and that feels like it's not specific to that 1 resource type (ie: AI thinks it happens for every resource vs a specific resources)
b
ai slop gonna slop. ¯\_(ツ)_/¯
b
indeed
b
I did tell it that it wasn't everything just some
Either way, you know the issue is somewhere within rancher
Open in Slack
PreviousNext
Powered by