This message was deleted.

Adding some logs: Logs Rancher Server

2024/06/04 17:15:32 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers fo │
│ r cluster c-m-jsx7jx2x: ClusterUnavailable 503: cluster not found, requeuing                                                                                                              │
│ 2024/06/04 17:17:32 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers fo │
│ r cluster c-m-jsx7jx2x: ClusterUnavailable 503: cluster not found, requeuing                                                                                                              │
│ W0604 17:18:22.952980      39 transport.go:301] Unable to cancel request for *client.addQuery                                                                                             │
│ W0604 17:18:22.953391      39 transport.go:301] Unable to cancel request for *client.addQuery

│ 2024/06/04 17:35:33 [ERROR] error syncing 'c-m-jsx7jx2x': handler cluster-deploy: cluster context c-m-jsx7jx2x is unavailable, requeuing                                                  │
│ 2024/06/04 17:35:33 [INFO] [planner] rkecluster fleet-default/cluster-a: configuring bootstrap node(s) custom-3290b4cb1b95: waiting for cluster agent to connect                          │
│ 2024/06/04 17:35:59 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers fo │
│ r cluster c-m-jsx7jx2x: ClusterUnavailable 503: cluster not found, requeuing                                                                                                              │
│ W0604 17:36:31.090158      39 warnings.go:80] <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3> MachineDeployment is deprecated; use <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1> MachineDeployment                                   │
│ W0604 17:37:22.310487      39 warnings.go:80] <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3> MachineSet is deprecated; use <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1> MachineSet                                                 │
│ 2024/06/04 17:37:27 [INFO] [planner] rkecluster fleet-default/cluster-a: configuring bootstrap node(s) custom-3290b4cb1b95: waiting for cluster agent to connect                          │
│ 2024/06/04 17:37:33 [ERROR] error syncing 'c-m-jsx7jx2x': handler cluster-deploy: cluster context c-m-jsx7jx2x is unavailable, requeuing                                                  │
│ 2024/06/04 17:37:33 [INFO] [planner] rkecluster fleet-default/cluster-a: configuring bootstrap node(s) custom-3290b4cb1b95: waiting for cluster agent to connect                          │
│ 2024/06/04 17:37:59 [ERROR] error syncing '_all_': handler user-controllers-controller: userControllersController: failed to set peers for key _all_: failed to start user controllers fo │
│ r cluster c-m-jsx7jx2x: ClusterUnavailable 503: cluster not found, requeuing                                                                                                              │
│ W0604 17:38:16.359467      39 warnings.go:80] <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3> MachineHealthCheck is deprecated; use <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1> MachineHealthCheck                                 │
│ 2024/06/04 17:38:38 [INFO] [planner] rkecluster fleet-default/cluster-a: configuring bootstrap node(s) custom-3290b4cb1b95: waiting for cluster agent to connect

Logs Fleet Controller:

│ time="2024-06-03T12:15:00Z" level=info msg="All controllers have been started"                                                                                                            │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=Role controller"                                                                                  │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://fleet.cattle.io/v1alpha1|fleet.cattle.io/v1alpha1>, Kind=ClusterRegistration controller"                                                                       │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://fleet.cattle.io/v1alpha1|fleet.cattle.io/v1alpha1>, Kind=ClusterRegistrationToken controller"                                                                  │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting /v1, Kind=ConfigMap controller"                                                                                                      │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://fleet.cattle.io/v1alpha1|fleet.cattle.io/v1alpha1>, Kind=Cluster controller"                                                                                   │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=RoleBinding controller"                                                                           │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://fleet.cattle.io/v1alpha1|fleet.cattle.io/v1alpha1>, Kind=GitRepoRestriction controller"                                                                        │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://fleet.cattle.io/v1alpha1|fleet.cattle.io/v1alpha1>, Kind=Bundle controller"                                                                                    │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=ClusterRoleBinding controller"                                                                    │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=ClusterRole controller"                                                                           │
│ time="2024-06-03T12:15:00Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"                                                                                                 │
│ time="2024-06-03T14:11:01Z" level=info msg="While calculating status.ResourceKey, error running helm template for bundle mcc-cluster-a-managed-system-upgrade-controller with target opti │
│ ons from : chart requires kubeVersion: >= 1.23.0-0 which is incompatible with Kubernetes v1.20.0"

Logs Capi Controller Manager

I0604 17:41:16.246263       1 machine_controller_phases.go:286] "Waiting for infrastructure provider to create machine infrastructure and report status.ready" controller="machine" contr │
│ ollerGroup="<http://cluster.x-k8s.io|cluster.x-k8s.io>" controllerKind="Machine" Machine="fleet-default/custom-b83dbd33166c" namespace="fleet-default" name="custom-b83dbd33166c" reconcileID=9a3ba2a5-8acb-44f7-b3 │
│ f1-581527addefa Cluster="fleet-default/cluster-a" Cluster="fleet-default/cluster-a" CustomMachine="fleet-default/custom-b83dbd33166c"                                                     │
│ I0604 17:41:16.246307       1 machine_controller_noderef.go:54] "Waiting for infrastructure provider to report spec.providerID" controller="machine" controllerGroup="<http://cluster.x-k8s.io|cluster.x-k8s.io>" c │
│ ontrollerKind="Machine" Machine="fleet-default/custom-b83dbd33166c" namespace="fleet-default" name="custom-b83dbd33166c" reconcileID=9a3ba2a5-8acb-44f7-b3f1-581527addefa Cluster="fleet- │
│ default/cluster-a" Cluster="fleet-default/cluster-a" CustomMachine="fleet-default/custom-b83dbd33166c"                                                                                    │
│ I0604 17:41:20.839385       1 machine_controller_phases.go:286] "Waiting for infrastructure provider to create machine infrastructure and report status.ready" controller="machine" contr │
│ ollerGroup="<http://cluster.x-k8s.io|cluster.x-k8s.io>" controllerKind="Machine" Machine="fleet-default/custom-b877043bafc2" namespace="fleet-default" name="custom-b877043bafc2" reconcileID=e4bc80ca-afe1-41e7-b7 │
│ 61-6808e56ba149 Cluster="fleet-default/cluster-a" Cluster="fleet-default/cluster-a" CustomMachine="fleet-default/custom-b877043bafc2"                                                     │
│ I0604 17:41:20.839420       1 machine_controller_noderef.go:54] "Waiting for infrastructure provider to report spec.providerID" controller="machine" controllerGroup="<http://cluster.x-k8s.io|cluster.x-k8s.io>" c │
│ ontrollerKind="Machine" Machine="fleet-default/custom-b877043bafc2" namespace="fleet-default" name="custom-b877043bafc2" reconcileID=e4bc80ca-afe1-41e7-b761-6808e56ba149 Cluster="fleet- │
│ default/cluster-a" Cluster="fleet-default/cluster-a" CustomMachine="fleet-default/custom-b877043bafc2"

Hey Paulo, I am facing a similar problem. Did you solve this in the mean time?

Hi @modern-winter-77378 I solved this problem by separating the ETCD from the Controlplane. This was necessary because there is a rule in Taint's Cloud Controller Provider that does not allow deployment on ETCD nodes. In my case I have a external Cloud Controller Provider.