Hi everyone. I have the following problem. I’m running Rancher 2.5.6 and Rancher 2.6.3 and getting the same error when trying to deploy an RKE cluster to our vSphere 7 environment. After the VM is created I’m getting the following error:

Error in driver during machine creation: 500 Internal Server Error, requeuing

The VM stays shut down and rancher tries again until I stop it. Has someone similar issues? Thanks in advance!

Hello, I’ve been handed the new requirement of updating all our virtual machines to ubuntu 22.04 lts. I’m trying to create a new k8s cluster via rancher/vSphere like normal, but I can’t get my ubuntu 22 template to provision. I’ve gotten past quite a few hurdles with it, and it’s really close, but is now failing here:

2022/05/18 14:43:19 [INFO] [node-controller-rancher-machine] Docker is up and running!
2022/05/18 14:43:19 [INFO] [node-controller-rancher-machine] (staging-worker-5) Closing plugin on server side
2022/05/18 14:43:19 [INFO] [node-controller-rancher-machine] (flag-lookup) Closing plugin on server side
2022/05/18 14:43:20 [INFO] Generating and uploading node config staging-worker-5
2022/05/18 14:43:20 [ERROR] [node] enqueing node pool c-b87mv:np-7wt7s
2022/05/18 14:43:20 [ERROR] error syncing 'c-b87mv/m-vglz6': handler node-controller: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I’ve googled around, and I can’t find much about this error, so I’m hoping someone can help me out, thanks in advance.

Ok, I have a really stupid question. How does rke run kubelet on the provisioned virtual machine? I’m having an issue with one node, rancher is complaining that kubelet isn’t reachable, and when I ssh into the node, kubelet isn’t running as a service. Yet, I do see a kubelet process if I run ps.

Hi all, I'd like to know if anyone knows how to import a downstream vsphere cluster into a another Rancher control plane and then delete it from the old one without actually deleting the nodes? I just tried this with a test cluster, there was no problem using the "import" feature and the downstream cluster appeared in the other Rancher Control plane. But of course the downstream cluster still exists in the old control plane and is completely deleted from vsphere when doing "delete". Has anyone tried this?

@brave-country-95392 you might need to be on vSphere 7 for that to work.

Hi, Has anyone had any success with NSX CNI in RKE2?

Hi, I’ve had success with Rancher on vSphere prior but now I receive an error saying “Unable to access the virtual machine configuration: Unable to access file [Datastore]“. I’m not sure what has changed as I was able to deploy clusters through this method before. For testing, the cloud credential user has full permissions to access all resources on vSphere. Any suggestions?

hi guys, i just noticed that when creating rancher rke2 that when it come to change the size of the pools hard disk and click save, it reverts back to the old value again. Whats the right way to change hard disk size or any resource?

If I have a RKE master VM that isn't booting/boot strapping, what's the best way to fix that? Just scale it down and build a new one?

@agreeable-oil-87482 i hope you don't mind me asking but i was watching your video on provisioning rancher on vsphere infrastructure. i am trying to emulate the way you assigned ip addresses with the network protocol profile but i wasn't quite sure if i got it right. i understand how you got the ip address metra data from the vaspp to the vm through cloud init but how does the ip get to the vapp /vm allocation? many thanks for your help

^ this is what is configured in the

/var/lib/cloud/instance/cloud-config.cfg

file

Is there a correct value to use in the Cloud Config YAML so that my Rancher provisioned vpshere VM gets the correct hostname? I’ve tried something like

local-hostname: {name}

and

hostname: {name}

in the cloud-config, and this seems to work as Rancher somehow knows to set the node name to whatever it’s being provisioned with, however my provisioned systems cloud-config gets a value that looks like:

local-hostname:
    name: null

or

hostname:
    name: null

Has anyone deployed from vsphere from a content library? what vsphere permissions are required for that?

How do you handle ip addressing for worker/control plane nodes? dhcp/dhcp reservations/ other?

Hi, trying to spin up a cluster on vsphere, has anyone got a server build which contains VMware tools we could use?

Hi we are building out a cluster on vsphere and we need the nodes to have static ips. What’s the best way to achieve this?

Hi, What do you think about using Vsphere Vsan as PV ? I'm using VMware ESXi, 7.0.3 with VC. and rancher 2.5.15

When I Rancher v2 provision a downstream rke2 cluster on vsphere, how can I securely get the vsphere-cpi-secret to the downstream cluster? I would like Rancher to put the secret in the kube-system namespace for me to address the security warning below extracted from the vsphere cpi chart readme: Warning: When the option to generate the Secret is enabled, the credentials are visible in the API to authorized users. If you create the Secret yourself they will not be visible. https://github.com/rancher/vsphere-charts/tree/main/charts/rancher-vsphere-cpi#installation

Is there a way to add the label config into the cpi/csi drivers installed with the vsphere cloud provider?

[Labels]
region = k8s-region
zone = k8s-zone

I have to do it manually in the configmap because it seems the charts are old and don't support the helm value options for that as I was hoping to add those values to the cluster definition where the helm options for the CPI and CSI providers are.

Hello, I’ve having some strange issues with my setup: • 3 microk8s nodes in ha • rancher-master deployed from helm on microk8s cluster • ingress-nginx for ingress with 3 pods • one vsphere cluster with vcenter • added vsphere cloud provider in rancher ui • created node template to deploy k8s in vsphere from rancher • The strange thing that i’m seeing is that when deploying a new k8s cluster, rancher keeps creating multiple vms on vcenter, even when deploying single node k8s it tries to bring up 2-4 nodes, then deletes 2-3 of then, keeping only one, eventually creating the one node k8s, but if i try to scale up(+1) it again tries to bring up 2-6 vms, and repeat, without finishing the scale up process • microk8s version: 1.24(ubuntu) • rancher-master helm chart version: 2.6.8(tried also 2.6.7) same stuff happens Did anyone happened to come across this strange issue when deploying to vsphere ? Is there something that i’m missing in the setup: helm install command: helm install rancher-master rancher-stable/rancher --set addLocal=“true” --set=ingress.enabled=“false” --namespace cattle-system --set hostname=controlplane.example.com --set replicas=3 --set tls=“external” --version 2.6.8

When setting up the dhcp, when getting a new network do we use the whole network within the scope or is there anything we need to set aside or beware off?

hi guys, i am trying to deploy rancher cluster to a seperate vlan. i had issues with downloading images but I solved that with a private registry mirror. now the deployment does not finish and its just sitting tthere. I am getting this in the rancher-system-agent. The bold part is most interesting, the rest are repeated. Any help to diagnose this issue? TIA i get this error: ct 24 200734 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200734+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 200739 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200739+13:00" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 150405 MST 2006\": cannot parse \"\" as \"Mon\"" Oct 24 200739 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200739+13:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2> Oct 24 200739 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200739+13:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler" Oct 24 200739 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200739+13:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt:> Oct 24 200739 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200739+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 150405 MST 2006\": cannot parse \"\" as \"Mon\"" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221024-200744/be7588c5c8d8b0831b57f1afe08907455a> Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0stdout] Name Location Size Created" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt:> Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2> Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt:> Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 200744 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T200744+13:00" level=error

I’m trying to create RKE cluster with the help of vSphere provider using a preconfigured node template in Rancher. When Rancher creates the VM nodes on VMware environment, I need them to have unique hostnames and IP’s. The problem is that the nodes are provisioned simultaneously from the template in vSphere and at first power-on, but they strangely get the same IP address! after investigating on VMs that is delpoyed from node template, they have different MAC but their IP addresses are the same. I need the nodes to be configured with unique IP’s. How do I fix this issue?

We have been using Rancher in a POC environment using VSphere. We followed this: https://www.rancher.co.jp/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/vsphere/ for 2.6 in regards to permissions. However when we try to build a downstream on Vsphere it sits on pending. When we get the Vsphere to have full access the issues goes away and we can build clusters. Is there any deviations from the documentation that we are missing? Many thanks for any replies.

Hello fellow ranchers. I have a problem with deploying vpsphere rke2 cluster. all goes well with the first server being created from a Rocky 8.5 template. but then it only stays in waiting to connect to cluster.

any help would be appreciated pointing me in the right direction to solve this.

apparently it had to do with vsphere as out of tree config.

I haven’t tested this yet, but my understanding is that with vSphere 7+ and RKE2+, the CSI out-of-tree provider is the only one supported. We are on vSphere 7, but using RKE1 clusters that were originally stood up in vSphere 6.7 - and we’re on the in-tree provider.

Is there an up to date guide for Rancher 2.6.9 to configure vSphere CSI in out of tree? The Guide for 2.5 mentions a plugin for the storage class but the plugin is no longer available. When creating the cluster I set the Cloud Provider to vSphere and add the Add-on Config for vSphere CPI Configuration I can see a storage class with the csi.vsphere.vmware.com Provisioner has been created. But when using it to make PVC it never becomes ready, did I miss something?

Hello