Hey Everyone, I hope all is well. I've been having a node provisioning issue since Rancher 2.7.6. I've upgraded to Rancher 2.8.2 and still have this issue. See the github link below, which I have commented in. Basically, any node I provision gets stuck in "Waiting for node ref". I've tried both Ubuntu 22.04 LTS and Rocky Linux 9 with the same result. https://github.com/rancher/rancher/issues/43701 Here are the logs from the rancher-system-agent on the machine i'm trying to provision

Feb 09 16:26:00 kube-dev-ctrl-01 rancher-system-agent[5946]: time="2024-02-09T16:26:00Z" level=info msg="Rancher System Agent version v0.3.3 (9e827a5) is starting"
Feb 09 16:26:00 kube-dev-ctrl-01 rancher-system-agent[5946]: time="2024-02-09T16:26:00Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Feb 09 16:26:00 kube-dev-ctrl-01 rancher-system-agent[5946]: time="2024-02-09T16:26:00Z" level=info msg="Starting remote watch of plans"
Feb 09 16:26:01 kube-dev-ctrl-01 rancher-system-agent[5946]: E0209 16:26:01.523825    5946 memcache.go:206] couldn't get resource list for <http://management.cattle.io/v3|management.cattle.io/v3>:
Feb 09 16:26:01 kube-dev-ctrl-01 rancher-system-agent[5946]: time="2024-02-09T16:26:01Z" level=info msg="Starting /v1, Kind=Secret controller"

And here are the logs from one of the Rancher pods.

2024/02/09 16:26:03 [INFO] [planner] rkecluster fleet-default/dev: waiting for at least one control plane, etcd, and worker node to be registered
2024/02/09 16:26:03 [ERROR] [rkebootstrap] fleet-default/custom-81bdf9ef4174: error getting machine by owner reference no matching controller owner ref
2024/02/09 16:26:03 [ERROR] error syncing 'fleet-default/custom-81bdf9ef4174': handler rke-bootstrap: no matching controller owner ref, requeuing
2024/02/09 16:26:03 [ERROR] [rkebootstrap] fleet-default/custom-81bdf9ef4174: error getting machine by owner reference no matching controller owner ref
2024/02/09 16:26:03 [ERROR] error syncing 'fleet-default/custom-81bdf9ef4174': handler rke-bootstrap: no matching controller owner ref, requeuing
2024/02/09 16:26:03 [ERROR] [rkebootstrap] fleet-default/custom-81bdf9ef4174: error getting machine by owner reference no matching controller owner ref
2024/02/09 16:26:03 [ERROR] error syncing 'fleet-default/custom-81bdf9ef4174': handler rke-bootstrap: no matching controller owner ref, requeuing
2024/02/09 16:26:04 [ERROR] [rkebootstrap] fleet-default/custom-81bdf9ef4174: error getting machine by owner reference no matching controller owner ref
2024/02/09 16:26:04 [ERROR] error syncing 'fleet-default/custom-81bdf9ef4174': handler rke-bootstrap: no matching controller owner ref, requeuing
2024/02/09 16:26:04 [ERROR] [rkebootstrap] fleet-default/custom-81bdf9ef4174: error getting machine by owner reference no matching controller owner ref
2024/02/09 16:26:04 [ERROR] error syncing 'fleet-default/custom-81bdf9ef4174': handler rke-bootstrap: no matching controller owner ref, requeuing
2024/02/09 16:26:04 [INFO] [planner] rkecluster fleet-default/dev: waiting for at least one control plane, etcd, and worker node to be registered