Any troubleshooting tips? Rancher logs are spamming this, where 10.1.0.11 is an IP of a controlplane node from the dev-cluster.

2023/11/03 20:33:50 [ERROR] Failed to handle tunnel request from remote address 10.0.1.155:35108 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:51 [ERROR] Failed to handle tunnel request from remote address 10.0.4.44:38834 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:51 [ERROR] Failed to handle tunnel request from remote address 10.0.1.155:42304 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:52 [ERROR] Failed to handle tunnel request from remote address 10.0.0.187:36348 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:55 [ERROR] Failed to handle tunnel request from remote address 10.0.4.44:60698 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:55 [ERROR] Failed to handle tunnel request from remote address 10.0.0.187:36374 (X-Forwarded-For: 10.1.0.11): response 400: cluster not found
2023/11/03 20:33:56 [INFO] [planner] rkecluster fleet-default/dev-cluster: configuring bootstrap node(s) custom-1fe1a39a3d72: waiting for probes: kube-controller-manager, kube-scheduler
2023/11/03 20:33:56 [INFO] [planner] rkecluster fleet-default/dev-cluster: configuring bootstrap node(s) custom-1fe1a39a3d72: waiting for probes: kube-controller-manager, kube-scheduler
2023/11/03 20:33:56 [ERROR] Failed to handle tunnel request from remote address 10.0.1.155:57670 (X-Forwarded-For: 10.1.0.11): response 400: cluster not found
2023/11/03 20:33:56 [ERROR] Failed to handle tunnel request from remote address 10.0.4.44:60712 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:56 [ERROR] Failed to handle tunnel request from remote address 10.0.4.44:60716 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:57 [ERROR] Failed to handle tunnel request from remote address 10.0.4.44:57574 (X-Forwarded-For: 10.1.0.11): response 401: failed authentication
2023/11/03 20:33:58 [ERROR] Failed to handle tunnel request from remote address 10.0.0.187:36990 (X-Forwarded-For: 10.1.0.11): response 400: cluster not found
2023/11/03 20:33:58 [ERROR] Failed to handle tunnel request from remote address 10.0.1.155:42320 (X-Forwarded-For: 10.1.0.11): response 400: cluster not found

All of the controlplane nodes in that cluster have the status

Waiting for probes: kube-controller-manager, kube-scheduler

Trying to provision a new rke2 node was failing, so rancher was updated to 2.7.9, but everything looks pretty much the same.

Nov 03 19:32:02 nodename rancher-system-agent[1923812]: E1103 19:32:02.758438 1923812 memcache.go:206] couldn't get resource list for <http://management.cattle.io/v3|management.cattle.io/v3>:
Nov 03 19:32:02 nodename rancher-system-agent[1923812]: time="2023-11-03T19:32:02Z" level=info msg="Starting /v1, Kind=Secret controller"
Nov 03 20:15:22 nodename rancher-system-agent[1923812]: W1103 20:15:22.676763 1923812 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 155; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
Nov 03 20:31:18 nodename rancher-system-agent[1923812]: W1103 20:31:18.995348 1923812 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 163; INTERNAL_ERROR; received from peer") has prevented the request from succeeding