Hey all! I was playing around harvester and seem to be stuck. I want to create a k8s cluster with the harvester nodedriver on a rancher installation (a VM inside harvester) . As soon as I want to create a new cluster, I always keep getting stuck at the node-provisioning. My Nodepool VMs are deployed on harvester but it seems like the cloud-init is not triggered. When I open the console of the k8s-nodes, i always just see the default installer UI. Tried with ubuntu22.04 and centos 7.2009. I had it running once, but then I had just 1 single harvester node. anyone else with this problem or can reproduce it? thx! ProvisionLog:

Waiting for viable init node

Some docker logs of rancher:

2022/06/08 16:25:23 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-dttv8
2022/06/08 16:25:23 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:23 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-dxlfr with role project-owner in namespace p-9nwqb
2022/06/08 16:25:24 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-dxlfr with role admin in namespace p-dttv8
2022/06/08 16:25:24 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-dxlfr with role project-owner in namespace p-dttv8
2022/06/08 16:25:24 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:24 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-m-wm9fhmsw
2022/06/08 16:25:24 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:24 [ERROR] Reconciler error: Operation cannot be fulfilled on <http://machinedeployments.cluster.x-k8s.io|machinedeployments.cluster.x-k8s.io> "harvester-k8s001-master": the object has been modified; please apply your changes to the latest version and try again
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:25 [ERROR] error syncing 'fleet-default/harvester-k8s001-master-33169394-62dds': handler machine-provision: no machine owner ref, requeuing
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:26 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:26 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:28 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:28 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:28:28 [ERROR] Unknown error: Operation cannot be fulfilled on <http://preferences.management.cattle.io|preferences.management.cattle.io> "last-visited": the object has been modified; please apply your changes to the latest version and try again

22.04 is not yet supported, but I’m not sure about that centos version. Which versions of rancher and harvester are you using? and are you deploying using rke1 or rke2? I use 20.04 regularly on v2.6.5 + 1.0.2 and am able to deploy rke1 and rke2 clusters.

I’m using 1.0.2 and v2.6.5. RKE2 Setup > v1.23.6+k3s1. Strange that I got it running once then. No idea how that went. I will just check it with ubuntu 20.04 then and give a feedback. I got rancher itself working in ubuntu 22.04 with this little hack: https://github.com/rancher/rancher/issues/36238#issuecomment-1141244691

good luck, let us know how it goes.

this is sadly what I get all the time. not sure what I’m missing.

could you try it with rke2 instead of k3s

nothing different here

oh, wait I think I see the issue. That’s an ISO you’re using, no? you should use .img or .qcow2.

yeah its an iso

try the official ubuntu 20.04 cloud image: https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

thx for the link, gonna give this a try

no problem!

I didn't think Rancher included ISOs when creating a cluster, it certainly didn't - will have to check

hey folks, I have very much same problem - Harvester up and running and linked to the Rancher. Once I create cluster I can see VMs created up and running in the Harvester. I'm using official Ubuntu20.04 cloudinit image, tried both RKE2/k3s configs and it's the same - stuck in " waiting for viable init node"