Hi, We have Rancher installed (helm chart), using gateway api as ingress to rancher server. We imported some k8s clusters in Rancher. Now, using a kubeconfig either directly to the k8s api or via Rancher (download kubeconfig) works fine. However kubectl exec won't work using the kubeconfig via rancher server (it does work with kubeconfig directly to the k8s api). In the verbose logging - using the Rancher kubeconfig: kubectl -v=10 -n test exec -it mypod -- /bin/sh

curl -v -XPOST  -H "User-Agent: kubectl/v1.26.6+rke2r1 (linux/amd64) kubernetes/11902a8" -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "Authorization: Bearer <masked>" '<https://RANCHERFQDN/k8s/clusters/c-m-abcdefg/api/v1/namespaces/test/pods/mypod/exec?command=%2Fbin%2Fsh&container=containername&stdin=true&stdout=true&tty=true>'
I0124 16:31:17.787224 1062701 round_trippers.go:495] HTTP Trace: DNS Lookup for RANCHERFQDN resolved to [{RANCHER-GATEWAYAPI-INGRESS-IP }]
I0124 16:31:17.788032 1062701 round_trippers.go:510] HTTP Trace: Dial to tcp:RANCHER-GATEWAYAPI-INGRESS-IP:443 succeed
I0124 16:31:17.791339 1062701 round_trippers.go:553] POST <https://RANCHERFQDN/k8s/clusters/c-m-abcdefg/api/v1/namespaces/test/pods/mypod/exec?command=%2Fbin%2Fsh&container=containername&stdin=true&stdout=true&tty=true> 403 Forbidden in 5 milliseconds
I0124 16:31:17.791352 1062701 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 5 ms
I0124 16:31:17.791358 1062701 round_trippers.go:577] Response Headers:
I0124 16:31:17.791367 1062701 round_trippers.go:580]     Date: Wed, 24 Jan 2024 15:31:17 GMT
I0124 16:31:17.791374 1062701 round_trippers.go:580]     Server: envoy
I0124 16:31:17.791381 1062701 round_trippers.go:580]     Content-Length: 0
I0124 16:31:17.791822 1062701 helpers.go:246] server response object: [{
  "metadata": {}
}]
Error from server:

Whereas using the direct kubeconfig gives:

I0124 16:54:06.238711 1065143 round_trippers.go:466] curl -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.26.6+rke2r1 (linux/amd64) kubernetes/11902a8" '<https://RANCHERFQDN/k8s/clusters/c-m-abcdefg/api/v1/namespaces/test/pods/mypod/exec?command=%2Fbin%2Fsh&container=containername&stdin=true&stdout=true&tty=true>'
I0124 16:54:06.239470 1065143 round_trippers.go:495] HTTP Trace: DNS Lookup for RANCHERFQDN resolved to [{RANCHER-GATEWAYAPI-INGRESS-IP }]
I0124 16:54:06.240118 1065143 round_trippers.go:510] HTTP Trace: Dial to tcp:RANCHER-GATEWAYAPI-INGRESS-IP:6443 succeed
I0124 16:54:06.274380 1065143 round_trippers.go:553] POST <https://RANCHERFQDN/k8s/clusters/c-m-abcdefg/api/v1/namespaces/test/pods/mypod/exec?command=%2Fbin%2Fsh&container=containername&stdin=true&stdout=true&tty=true> 101 Switching Protocols in 35 milliseconds
I0124 16:54:06.274580 1065143 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 35 ms
I0124 16:54:06.274645 1065143 round_trippers.go:577] Response Headers:
I0124 16:54:06.274696 1065143 round_trippers.go:580]     Connection: Upgrade
I0124 16:54:06.274779 1065143 round_trippers.go:580]     Upgrade: SPDY/3.1
I0124 16:54:06.274960 1065143 round_trippers.go:580]     X-Stream-Protocol-Version: v4.channel.k8s.io

The difference here is the response header upgrade being upgraded, when going directly to the k8s api. How can that be achieved using the rancher kubeconfig (via the gateway api)?