Hi all, I am trying to deploy an RKE2 cluster on vSphere via the Rancher UI. I use the below details. • Rancher Helm installation 2.7.5-stable • vSphere Client version 7.0.3.00700 As a basic step, I tried to deploy a basic RKE2 cluster with 1 node using the cloud-provider vSphere, Canal as my CNI and version 1.26.6+rke2r1 As for the VM template I used a basic 22.04 Ubuntu image and I installed all the needed dependencies based on the following guide: https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template It's worth mentioning that ufw was disabled for testing purposes. Whatever I tried, the cluster is not getting created by instead is stuck in "waiting for agent to check in and apply initial plan" state. The logs from the Rancher pod can be found below.

2023/08/03 06:50:39 [ERROR] [rkecluster] fleet-default/rke2-canal: error getting CAPI cluster no matching controller owner ref
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/03 06:50:39 [ERROR] [rkecluster] fleet-default/rke2-canal: error getting CAPI cluster no matching controller owner ref
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/03 06:50:39 [ERROR] [planner] rkecluster fleet-default/rke2-canal: error encountered during plan processing was no matching controller owner ref
2023/08/03 06:50:39 [ERROR] [rkecluster] fleet-default/rke2-canal: error getting CAPI cluster no matching controller owner ref
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/03 06:50:39 [ERROR] [rkecluster] fleet-default/rke2-canal: error getting CAPI cluster no matching controller owner ref
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/03 06:50:39 [ERROR] [rkecluster] fleet-default/rke2-canal: error getting CAPI cluster no matching controller owner ref
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler planner: no matching controller owner ref, requeuing
2023/08/03 06:50:39 [ERROR] [planner] rkecluster fleet-default/rke2-canal: error encountered during plan processing was no matching controller owner ref
2023/08/03 06:50:39 [ERROR] [rkecluster] fleet-default/rke2-canal: error getting CAPI cluster no matching controller owner ref
2023/08/03 06:50:39 [ERROR] error syncing 'fleet-default/rke2-canal': handler rke-cluster: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [planner] rkecluster fleet-default/rke2-canal: error encountered during plan processing was no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal': handler planner: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [planner] rkecluster fleet-default/rke2-canal: error encountered during plan processing was no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal': handler planner: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [planner] rkecluster fleet-default/rke2-canal: error encountered during plan processing was no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal': handler planner: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for infrastructure ready
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for infrastructure ready
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for infrastructure ready
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for infrastructure ready
2023/08/03 06:50:40 [ERROR] [CAPI] Reconciler error: Operation cannot be fulfilled on <http://machinedeployments.cluster.x-k8s.io|machinedeployments.cluster.x-k8s.io> "rke2-canal-pool1": the object has been modified; please apply your changes to the latest version and try again
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for infrastructure ready
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/03 06:50:40 [ERROR] [rkebootstrap] fleet-default/rke2-canal-bootstrap-template-jw4bv: error getting machine by owner reference no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-bootstrap-template-jw4bv': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [rkebootstrap] fleet-default/rke2-canal-bootstrap-template-jw4bv: error getting machine by owner reference no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-bootstrap-template-jw4bv': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: error getting machine by owner reference: no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt': handler machine-provision: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: error getting machine by owner reference: no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt': handler machine-provision: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [rkebootstrap] fleet-default/rke2-canal-bootstrap-template-jw4bv: error getting machine by owner reference no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-bootstrap-template-jw4bv': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: error getting machine by owner reference: no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt': handler machine-provision: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [rkebootstrap] fleet-default/rke2-canal-bootstrap-template-jw4bv: error getting machine by owner reference no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-bootstrap-template-jw4bv': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: error getting machine by owner reference: no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt': handler machine-provision: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for at least one control plane, etcd, and worker node to be registered
2023/08/03 06:50:40 [ERROR] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: error getting machine by owner reference: no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt': handler machine-provision: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [rkebootstrap] fleet-default/rke2-canal-bootstrap-template-jw4bv: error getting machine by owner reference no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-bootstrap-template-jw4bv': handler rke-bootstrap: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [ERROR] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: error getting machine by owner reference: no matching controller owner ref
2023/08/03 06:50:40 [ERROR] error syncing 'fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt': handler machine-provision: no matching controller owner ref, requeuing
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] EnsureSecretForServiceAccount: waiting for secret [rke2-canal-bootstrap-template-jw4bv-machine-bootstrapz5jv] to be populated with token
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:40 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:41 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:41 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:41 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:41 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:49 [INFO] [planner] rkecluster fleet-default/rke2-canal: waiting: waiting for viable init node
2023/08/03 06:50:50 [INFO] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: reconciling machine job
2023/08/03 06:50:50 [INFO] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: reconciling machine job
2023/08/03 06:50:50 [INFO] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: reconciling machine job
2023/08/03 06:50:50 [INFO] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: reconciling machine job
2023/08/03 06:50:50 [INFO] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: reconciling machine job
2023/08/03 06:50:50 [INFO] [machineprovision] fleet-default/rke2-canal-pool1-bb5a7e33-qb9zt: reconciling machine job