also, are you getting anything after "Error from server:", like a stack trace or something?

the only different is that I use

istio

as ingress gateway and I just create my cert, virtual-service and gateway apart from the chart.

apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Certificate
metadata:
  name: <http://my-rancher.test.example.com|my-rancher.test.example.com>
  namespace: istio-system
spec:
  privateKey:
    rotationPolicy: Always
  secretName: <http://my-rancher.test.example.com|my-rancher.test.example.com>
  commonName: <http://my-rancher.test.example.com|my-rancher.test.example.com>
  issuerRef:
    name: letsencrypt-prod-istio
    kind: ClusterIssuer
  dnsNames:
  - <http://my-rancher.test.example.com|my-rancher.test.example.com>
---
apiVersion: <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
kind: Gateway
metadata:
  name: rancher
  namespace: cattle-system
spec:
  selector:
    app: istio-ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
      - <http://my-rancher.test.example.com|my-rancher.test.example.com>
      tls:
        mode: SIMPLE
        credentialName: <http://my-rancher.test.example.com|my-rancher.test.example.com>
---
apiVersion: <http://networking.istio.io/v1beta1|networking.istio.io/v1beta1>
kind: VirtualService
metadata:
  name: rancher
  namespace: cattle-system
spec:
  gateways:
  - rancher
  hosts:
  - <http://my-rancher.test.example.com|my-rancher.test.example.com> 
  http:
  - name: "http"
    route:
    - destination:
        host: rancher.cattle-system.svc.cluster.local
        port:
          number: 80

I found a github issue with others having the same issue but it seems there is no response: https://github.com/rancher/rancher/issues/37355

For some reason my request gets denied with

403

Forbidden:

curl -k -v -XPOST  -H "X-Stream-Protocol-Version: <http://v4.channel.k8s.io|v4.channel.k8s.io>" -H "X-Stream-Protocol-Version: <http://v3.channel.k8s.io|v3.channel.k8s.io>" -H "X-Stream-Protocol-Version: <http://v2.channel.k8s.io|v2.channel.k8s.io>" -H "X-Stream-Protocol-Version: <http://channel.k8s.io|channel.k8s.io>" -H "User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba" -H "Authorization: Bearer <masked>" '<https://my-rancher.test.example.com/k8s/clusters/local/api/v1/namespaces/cattle-system/pods/rancher-58f6c44c59-zg2ft/exec?command=bash&container=rancher&stdin=true&stdout=true&tty=true>'
I1028 11:32:40.264117   42985 round_trippers.go:454] POST <https://my-rancher.test.example.com/k8s/clusters/local/api/v1/namespaces/cattle-system/pods/rancher-58f6c44c59-zg2ft/exec?command=bash&container=rancher&stdin=true&stdout=true&tty=true> 403 Forbidden in 87 milliseconds
I1028 11:32:40.264139   42985 round_trippers.go:460] Response Headers:
I1028 11:32:40.264145   42985 round_trippers.go:463]     Server: istio-envoy
I1028 11:32:40.264150   42985 round_trippers.go:463]     Content-Length: 0
I1028 11:32:40.264153   42985 round_trippers.go:463]     Date: Fri, 28 Oct 2022 09:32:39 GMT
I1028 11:32:40.264786   42985 helpers.go:216] server response object: [{
  "metadata": {}
}]
F1028 11:32:40.264859   42985 helpers.go:115] Error from server:
F1028 11:32:40.264859   42985 helpers.go:115] Error from server:
goroutine 1 [running]:
<http://k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00000e001|k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00000e001>, 0xc000f96280, 0x42, 0x133)
*** TRUNCATED

It seems that Envoy does not support the SPDY protocol. so that explain why it fails with

exec

and

port-forward

. I could then solve it If the Rancher Pods load my SSL certificate instead of the

Self-signed

one. Then I cann just

passthrough

and make the SSL termination directly on the Rancher Pod with my trusted certificate. Unfortunately I have not found how to achieve that, or if that is possible. any ideas would be extremely appreciated !