https://rancher.com/ logo
h

hundreds-waiter-30127

06/08/2022, 4:33 PM
Hey all! I was playing around harvester and seem to be stuck. I want to create a k8s cluster with the harvester nodedriver on a rancher installation (a VM inside harvester) . As soon as I want to create a new cluster, I always keep getting stuck at the node-provisioning. My Nodepool VMs are deployed on harvester but it seems like the cloud-init is not triggered. When I open the console of the k8s-nodes, i always just see the default installer UI. Tried with ubuntu22.04 and centos 7.2009. I had it running once, but then I had just 1 single harvester node. anyone else with this problem or can reproduce it? thx! ProvisionLog:
Waiting for viable init node
Some docker logs of rancher:
Copy code
2022/06/08 16:25:23 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace p-dttv8
2022/06/08 16:25:23 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:23 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-dxlfr with role project-owner in namespace p-9nwqb
2022/06/08 16:25:24 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-dxlfr with role admin in namespace p-dttv8
2022/06/08 16:25:24 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-dxlfr with role project-owner in namespace p-dttv8
2022/06/08 16:25:24 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:24 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-m-wm9fhmsw
2022/06/08 16:25:24 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:24 [ERROR] Reconciler error: Operation cannot be fulfilled on <http://machinedeployments.cluster.x-k8s.io|machinedeployments.cluster.x-k8s.io> "harvester-k8s001-master": the object has been modified; please apply your changes to the latest version and try again
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:25 [ERROR] error syncing 'fleet-default/harvester-k8s001-master-33169394-62dds': handler machine-provision: no machine owner ref, requeuing
2022/06/08 16:25:25 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:26 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:26 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:27 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:28 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:25:28 [INFO] [planner] rkecluster fleet-default/harvester-k8s001: waiting: waiting for viable init node
2022/06/08 16:28:28 [ERROR] Unknown error: Operation cannot be fulfilled on <http://preferences.management.cattle.io|preferences.management.cattle.io> "last-visited": the object has been modified; please apply your changes to the latest version and try again
s

square-orange-60123

06/08/2022, 6:51 PM
22.04 is not yet supported, but I’m not sure about that centos version. Which versions of rancher and harvester are you using? and are you deploying using rke1 or rke2? I use 20.04 regularly on v2.6.5 + 1.0.2 and am able to deploy rke1 and rke2 clusters.
h

hundreds-waiter-30127

06/08/2022, 7:20 PM
I’m using 1.0.2 and v2.6.5. RKE2 Setup > v1.23.6+k3s1. Strange that I got it running once then. No idea how that went. I will just check it with ubuntu 20.04 then and give a feedback. I got rancher itself working in ubuntu 22.04 with this little hack: https://github.com/rancher/rancher/issues/36238#issuecomment-1141244691
👍 1
s

square-orange-60123

06/08/2022, 7:22 PM
good luck, let us know how it goes.
👍 1
h

hundreds-waiter-30127

06/08/2022, 7:37 PM
this is sadly what I get all the time. not sure what I’m missing.
s

square-orange-60123

06/08/2022, 8:09 PM
could you try it with rke2 instead of k3s
h

hundreds-waiter-30127

06/08/2022, 10:05 PM
nothing different here
s

square-orange-60123

06/08/2022, 10:06 PM
oh, wait I think I see the issue. That’s an ISO you’re using, no? you should use .img or .qcow2.
h

hundreds-waiter-30127

06/08/2022, 10:09 PM
yeah its an iso
s

square-orange-60123

06/08/2022, 10:09 PM
h

hundreds-waiter-30127

06/08/2022, 10:12 PM
thx for the link, gonna give this a try
👍 1
@square-orange-60123 that was it!
deployed centos with a qcow2 cloudimg and the clusters is building! awesome. Thanks for the hint!
s

square-orange-60123

06/08/2022, 10:35 PM
no problem!
🙌 1
w

witty-jelly-95845

06/09/2022, 1:36 PM
I didn't think Rancher included ISOs when creating a cluster, it certainly didn't - will have to check
1
i

incalculable-nail-18520

06/11/2022, 3:05 AM
hey folks, I have very much same problem - Harvester up and running and linked to the Rancher. Once I create cluster I can see VMs created up and running in the Harvester. I'm using official Ubuntu20.04 cloudinit image, tried both RKE2/k3s configs and it's the same - stuck in " waiting for viable init node"
Harvester seems VMs running :
12 Views