https://rancher.com/ logo
Title
q

quaint-alarm-7893

03/23/2023, 4:46 AM
i'm trying to get a custom rke2 or k3s cluster working on-top of harvester. but i'm having issues getting the cloud-provider working. i followed: https://docs.harvesterhci.io/v1.1/rancher/cloud-provider if i deploy a 1.24.4 RKE2 cluster with Harvester for the cloud provider, the cluster never finishes spinning up. if i try to do a custom RKE2/K3S cluster with no provider, then add the Harvester-Cloud-provider later, i get this error: Error getting instance metadata for node addresses: Get
"<https://192.168.5.100/k8s/clusters/local/apis/kubevirt.io/v1/namespaces/k8s/virtualmachines/k3s-m1>": x509: certificate signed by unknown authority
i imagine because my harvester cluster has a self-signed cert. any ideas on how to fix this? seems like my best bet is a custom-cluster for the provider, but i have no idea how to get the k3s or rke2 guest k8s cluster to trust harvester vip
f

full-plastic-79795

03/23/2023, 5:51 AM
Thank you for bringing this issue to our attention. Based on our investigation, it appears that the user may have been using a kubeconfig downloaded from the UI, rather than obtaining it from the /etc/rancher/rke2/rke2.yaml file after SSH into the Harvester node. Using a kubeconfig downloaded from the UI can cause this problem. We recommend that the user try obtaining the kubeconfig from within the node instead. Please let us know if this resolves the issue. Thank you.
q

quaint-alarm-7893

03/23/2023, 5:52 AM
are you talking about the kubeconfig used with the generate_addon.sh script?
f

full-plastic-79795

03/23/2023, 5:52 AM
Yes
q

quaint-alarm-7893

03/23/2023, 5:53 AM
great! i'll try that. thanks!
is it just me or can you not use the harvester provider w/ a custom rke2 cluster? it seems like every time i try that method, it just hangs on the install
f

full-plastic-79795

03/23/2023, 6:15 AM
Could you describe the step to reproduce in detail?
q

quaint-alarm-7893

03/23/2023, 6:16 AM
sure. standby. 🙂
🙏 1
ignore this one.
In Rancher Cluster Management click Create Cluster Pick “Custom” for the provider: Set cluster version to v1.24.4, because that’s what all the docs show. Newer versions seem to exhibit the same behavior. Click “Create” Grab the Registration command and save it. curl --insecure -fL https://{rancher-url} /system-agent-install.sh | sudo sh -s - --server https://{rancher-url} --label 'cattle.io/os=linux' --token {token} --ca-checksum {ca-checksum} --etcd --controlplane –worker Create a VM in harvester. I used a Ubuntu Cloud-img. Let me know if you’d like the YAML login to the VM once it’s ready, and run the curl command from the Registration Page from your Custom RKE2 Cluster. Watch logs with: sudo journalctl -u rke2-server -f Most of my clusters take about 8 minutes to stand up. I can let this run over-night and it never finishes.
👍 1
image.png
logs.txt
looks like it might have to do w/ it not creating the cloud-config files here too. similar issue as to when you do not select harvester as the provider, and you have to add it manually.
f

full-plastic-79795

03/23/2023, 6:42 AM
Thank you for providing us the steps. After reviewing your scenario, we have found that it is not covered in our current testing procedures, as we typically use the Harvester node driver to automatically create clusters.
q

quaint-alarm-7893

03/23/2023, 6:43 AM
which makes sense, but i'm trying to get a windows node up and running, that's why i'm running into this. seems the best way to get a windows node is the custom cluster, w/ the windows registration command (after bootstrapping your cluster w/ atleast 1 linux master)
do you guys have anything you are doing currently to get a windows node working with a rke2 guest-cluster?
f

full-plastic-79795

03/23/2023, 6:53 AM
I appreciate your question, but unfortunately, I don't have the relevant experience to provide you with a satisfactory answer. If you don't mind, could you please submit your question as a GitHub issue in harvester/harvester repo ? Please let me know if you have any further questions or concerns.
q

quaint-alarm-7893

03/23/2023, 7:24 AM
will do. thanks!
👍 1
@full-plastic-79795 and @salmon-city-57654 i created a custom cluster, dropped the new cloud-config generated from the rke.yaml file off one of the harvester servers onto the node, added the helm chart for the Harvester-CSI, and created a PVC. i get this error now:
Type     Reason                Age               From                                                                                                           Message
  ----     ------                ----              ----                                                                                                           -------
  Normal   Provisioning          4s (x7 over 68s)  driver.harvesterhci.io_harvester-csi-driver-controllers-779c557d47-srfwv_6e6919cb-2c85-4eef-ab22-88caf890c870  External provisioner is provisioning volume for claim "natimark/sync-pvc"
  Warning  ProvisioningFailed    4s (x7 over 67s)  driver.harvesterhci.io_harvester-csi-driver-controllers-779c557d47-srfwv_6e6919cb-2c85-4eef-ab22-88caf890c870  failed to provision volume with StorageClass "harvester": rpc error: code = Internal desc = persistentvolumeclaims is forbidden: User "system:serviceaccount:k8s:default" cannot create resource "persistentvolumeclaims" in API group "" in the namespace "k8s"
  Normal   ExternalProvisioning  1s (x6 over 68s)  persistentvolume-controller                                                                                    waiting for a volume to be created, either by external provisioner "<http://driver.harvesterhci.io|driver.harvesterhci.io>" or manually created by system administrator
that's when describing the PVC. it's a new error, so PROGRESS! 😄 any thoughts?
also worth noting, i'm using the service_account_name default, and namespace k8s for generate_addon.sh. i'm not sure if i should be maybe passing it something else? and do i have to do anything to create the service account first maybe?
another note. if i create a cluster w/ harvester as the cloud provider, grab that cloud-config, and use it in my custom cluster, it seems to provision the pvc just fine. so it must have something to do w/ that config generation.