Hi all, I am a developer in open-source project Sylva under the Linux Foundation (link to the project Sylva-projects · GitLab). Right now, we support the automation of the deployment of a Kubernetes cluster in a VSphere environment (i.e. in a VCenter) using ClusterAPI vsphere infrastructure provider and ClusterAPI Kubeadm provider. I am trying to use RKE2 as Kubernetes distribution instead of Kubeadm, and to do so I am using the ClusterAPI provider RKE2. The cluster I am creating has one control plane node and 3 worker nodes, all deployed in a VCenter as VSphere VMs. The VM and the control plane node comes up and works well, but I am facing some issues with the worker nodes. Specifically, worker nodes' VMs are not created, and I am stuck with this error in the rke2-control-plane-controller pod's log (I'm pasting here the part of the log that involves the error):

I0606 09:00:08.056514       1 rke2controlplane_controller.go:259]  "msg"="Ready Machine :" "RKE2ControlPlane"={"name":"management-cluster-control-plane","namespace":"default"} "controller"="rke2controlplane" "controllerGroup"="<http://controlplane.cluster.x-k8s.io|controlplane.cluster.x-k8s.io>" "controllerKind"="RKE2ControlPlane" "machine-name"="management-cluster-control-plane-kgnrr" "name"="management-cluster-control-plane" "namespace"="default" "reconcileID"="365d1f12-30b5-4e65-bc81-56669f968632"

E0606 09:00:08.062775       1 rke2controlplane_controller.go:152]  "msg"="Failed to update RKE2ControlPlane Status" "error"="some Control Plane machines exist and are ready but they have no IP Address available" "RKE2ControlPlane"={"name":"management-cluster-control-plane","namespace":"default"} "cluster"="management-cluster" "controller"="rke2controlplane" "controllerGroup"="<http://controlplane.cluster.x-k8s.io|controlplane.cluster.x-k8s.io>" "controllerKind"="RKE2ControlPlane" "name"="management-cluster-control-plane" "namespace"="default" "reconcileID"="365d1f12-30b5-4e65-bc81-56669f968632"

E0606 09:00:08.063894       1 controller.go:326]  "msg"="Reconciler error" "error"="some Control Plane machines exist and are ready but they have no IP Address available" "RKE2ControlPlane"={"name":"management-cluster-control-plane","namespace":"default"} "controller"="rke2controlplane" "controllerGroup"="<http://controlplane.cluster.x-k8s.io|controlplane.cluster.x-k8s.io>" "controllerKind"="RKE2ControlPlane" "name"="management-cluster-control-plane" "namespace"="default" "reconcileID"="365d1f12-30b5-4e65-bc81-56669f968632"

From the code in the RKE2 controPlane controller, it seems that it tries to retrieve a valid IP address for the ready control plane node, but the results is that it doesn't find any valid IP address. The code checks for both InternalIP and ExternalIP in the Machine CRD resource, I report here the Machine CRD I have for the control plane node:

Name:         management-cluster-control-plane-kgnrr

Namespace:    default

Labels:       <http://cluster.x-k8s.io/cluster-name=management-cluster|cluster.x-k8s.io/cluster-name=management-cluster>

<http://cluster.x-k8s.io/control-plane=|cluster.x-k8s.io/control-plane=>

<http://node.cluster.x-k8s.io/esxi-host=|node.cluster.x-k8s.io/esxi-host=><HOST IP ADDRESS>

Annotations:  <http://controlplane.cluster.x-k8s.io/rke2-server-configuration|controlplane.cluster.x-k8s.io/rke2-server-configuration>: {"disableComponents":{},"cni":"calico","etcd":{"backupConfig":{}}}

API Version:  <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>

Kind:         Machine

Metadata:

Creation Timestamp:  2023-06-01T14:49:37Z

Finalizers:

<http://machine.cluster.x-k8s.io|machine.cluster.x-k8s.io>

Generation:  3

Managed Fields:

API Version:  <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>

Fields Type:  FieldsV1

fieldsV1:

f:metadata:

f:annotations:

.:

f:<http://controlplane.cluster.x-k8s.io/rke2-server-configuration|controlplane.cluster.x-k8s.io/rke2-server-configuration>:

f:finalizers:

.:

v:"<http://machine.cluster.x-k8s.io|machine.cluster.x-k8s.io>":

f:labels:

.:

f:<http://cluster.x-k8s.io/cluster-name|cluster.x-k8s.io/cluster-name>:

f:<http://cluster.x-k8s.io/control-plane|cluster.x-k8s.io/control-plane>:

f:<http://node.cluster.x-k8s.io/esxi-host|node.cluster.x-k8s.io/esxi-host>:

f:ownerReferences:

.:

k:{"uid":"8080a6c3-49db-4bd0-bb81-60d475f98d6c"}:

f:spec:

.:

f:bootstrap:

.:

f:configRef:

f:dataSecretName:

f:clusterName:

f:infrastructureRef:

f:providerID:

f:version:

Manager:      manager

Operation:    Update

Time:         2023-06-01T14:51:03Z

API Version:  <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>

Fields Type:  FieldsV1

fieldsV1:

f:status:

.:

f:addresses:

f:bootstrapReady:

f:conditions:

f:infrastructureReady:

f:lastUpdated:

f:observedGeneration:

f:phase:

Manager:      manager

Operation:    Update

Subresource:  status

Time:         2023-06-01T14:58:20Z

Owner References:

API Version:           <http://controlplane.cluster.x-k8s.io/v1alpha1|controlplane.cluster.x-k8s.io/v1alpha1>

Block Owner Deletion:  true

Controller:            true

Kind:                  RKE2ControlPlane

Name:                  management-cluster-control-plane

UID:                   8080a6c3-49db-4bd0-bb81-60d475f98d6c

Resource Version:        4473

UID:                     08fc8851-837c-453f-8db8-d7bff8d4a6c7

Spec:

Bootstrap:

Config Ref:

API Version:     <http://bootstrap.cluster.x-k8s.io/v1alpha1|bootstrap.cluster.x-k8s.io/v1alpha1>

Kind:            RKE2Config

Name:            management-cluster-control-plane-tb6gk

Namespace:       default

UID:             38419d0a-d19e-4596-b827-a5107d19baab

Data Secret Name:  management-cluster-control-plane-tb6gk

Cluster Name:        management-cluster

Infrastructure Ref:

API Version:          <http://infrastructure.cluster.x-k8s.io/v1beta1|infrastructure.cluster.x-k8s.io/v1beta1>

Kind:                 VSphereMachine

Name:                 management-cluster-control-plane-jkmhg

Namespace:            default

UID:                  91393bdf-812b-43d9-9f54-8880cc78fb08

Node Deletion Timeout:  10s

Provider ID:            <vsphere://42351d94-6626-4ddb-ed61-5559bd9ffa8b>

Version:                v1.24.12+rke2r1

Status:

Addresses:

Address:        <IP ADDRESS>

Type:           ExternalIP

Bootstrap Ready:  true

Conditions:

Last Transition Time:  2023-06-01T14:50:53Z

Status:                True

Type:                  Ready

Last Transition Time:  2023-06-01T14:49:37Z

Status:                True

Type:                  BootstrapReady

Last Transition Time:  2023-06-01T14:50:53Z

Status:                True

Type:                  InfrastructureReady

Last Transition Time:  2023-06-01T14:58:20Z

Reason:                NodeProvisioning

Severity:              Warning

Status:                False

Type:                  NodeHealthy

Infrastructure Ready:    true

Last Updated:            2023-06-01T14:51:03Z

Observed Generation:     3

Phase:                   Provisioned

Events:                    <none>

For security reasons, I substituted the IP address with "<IP ADDRESS>" you see in "Address" field. This IP is assigned by a DHCP server, and it should be the one that the controller picks up for the control plane node, but this does not happen and I got the error mentioned above. May it be related to an implementation problem? Or is there a specific configuration needed that I am missing? Thanks in advance for the answer