hi , we have a legacy Rancher Cluster running Rancher version v2.4.4 Rke version v1.0.8 Kubernetes version v1.17.5 Docker version v18.9.2 Our deployments are failing with following errors

"unable to decode an event from the watch stream: backed up reader") has prevented the request from succeeding




[main] 2022/11/23 14:38:18 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:38:18 GRPC listening on :61000

[main] 2022/11/23 14:38:18 Probes listening on :50491

[main] 2022/11/23 14:38:18 Storage driver is ConfigMap

[main] 2022/11/23 14:38:18 Max history per release is 10

[tiller] 2022/11/23 14:38:19 getting history for release ne-campaign

[storage] 2022/11/23 14:38:19 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:38:19 preparing update for ne-campaign

[storage] 2022/11/23 14:38:19 getting deployed releases from "ne-campaign" history

[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:38:19 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:38:19 performing force update for ne-campaign

[storage] 2022/11/23 14:38:19 getting last revision of "ne-campaign"

[storage] 2022/11/23 14:38:19 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address





[main] 2022/11/23 14:38:19 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:38:19 GRPC listening on :50491

[main] 2022/11/23 14:38:19 Probes listening on :50491

[main] 2022/11/23 14:38:19 Storage driver is ConfigMap

[main] 2022/11/23 14:38:19 Max history per release is 10

[main] 2022/11/23 14:38:19 Probes server died: listen tcp :50491: bind: address already in use

[main] 2022/11/23 14:48:20 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:48:20 GRPC listening on :50491

[main] 2022/11/23 14:48:20 Probes listening on :61000

[main] 2022/11/23 14:48:20 Storage driver is ConfigMap

[main] 2022/11/23 14:48:20 Max history per release is 10

[tiller] 2022/11/23 14:48:21 getting history for release ne-campaign

[storage] 2022/11/23 14:48:21 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:21 preparing update for ne-campaign

[storage] 2022/11/23 14:48:21 getting deployed releases from "ne-campaign" history

[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:21 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:21 performing force update for ne-campaign

[storage] 2022/11/23 14:48:21 getting last revision of "ne-campaign"

[storage] 2022/11/23 14:48:21 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[main] 2022/11/23 14:48:22 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:48:22 GRPC listening on :50491

[main] 2022/11/23 14:48:22 Probes listening on :61000

[main] 2022/11/23 14:48:22 Storage driver is ConfigMap

[main] 2022/11/23 14:48:22 Max history per release is 10

[tiller] 2022/11/23 14:48:23 getting history for release ne-campaign

[storage] 2022/11/23 14:48:23 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:23 preparing update for ne-campaign

[storage] 2022/11/23 14:48:23 getting deployed releases from "ne-campaign" history

[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:23 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:23 performing force update for ne-campaign

[storage] 2022/11/23 14:48:23 getting last revision of "ne-campaign"

[storage] 2022/11/23 14:48:23 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

We have tried restarting the master nodes (& docker service ) on downstream Kubernetes nodes .. but same has not helped. We have tried deploying latest code on other micro services hosted on the same cluster but that is also failing. This error seems to have worsened now .. PFB the recent logs where its getting blocked at even earlier stage.

2022/11/28 14:40:29 [ERROR] Error certs=InvalidBodyContent 422: failed to find cert that matched private key parsing cert template-repository-prod-tls. Will not display correctly in UI

2022/11/28 14:43:12 [INFO] Installing chart using helm version: rancher-helm

[main] 2022/11/28 14:43:12 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/28 14:43:12 GRPC listening on :61000

[main] 2022/11/28 14:43:12 Probes listening on :61000

[main] 2022/11/28 14:43:12 Storage driver is ConfigMap

[main] 2022/11/28 14:43:12 Max history per release is 10

[main] 2022/11/28 14:43:12 Probes server died: listen tcp :61000: bind: address already in use

UPGRADE FAILED

Error: context deadline exceeded

2022/11/28 14:53:12 [ERROR] AppController p-746c7/ecomm-support [helm-controller] failed with : failed to install app ecomm-support. Error: UPGRADE FAILED: context deadline exceeded