Hello i am new to the community Please if you can help me wh rancher-users #vsphere

Title

# vsphere

hundreds-twilight-31144

02/08/2023, 6:54 AM

Hello, i am new to the community. Please if you can help me where to look. Install rancher 2.7.1 on ks3 1.24 on vsphere 7.0u3. Problem is that i am unable to create RKE cluster. DHCP is not enabled. The only message that i got timeout ssh key. In logs of vcentar api i see that user is logged in but dont see any other action like create vm from template. I try different templates and different variations and everytime message is the same. I stuck with this error for 10 days..

I try David repo all ubuntu variations with packer, downloading cloud img. We stuck on pre create checks and dont see that vcentar is generating vm from template that we prepare. Credentials are ok. Vcentar and rancher are in different vlans

Just to mention that rancher is installed on single node using helm

agreeable-oil-87482

02/08/2023, 9:12 AM

So you see the user logged into vcenter that you're using in your cloud credential in Rancher but no vm cloning activity? • Have you checked the vcenter server vpxd logs for that user? • Have you tried logging into vcenter with the these user credentials to verify it has access to the vm templates etc?

hundreds-twilight-31144

02/08/2023, 10:04 AM

in rancher logs

2023/02/08 100116 [INFO] [node-controller] Creating CA: /management-state/node/nodes/vafnode1/certs/ca.pem 2023/02/08 100116 [INFO] [node-controller] Creating CA: /management-state/node/nodes/vafnode3/certs/ca.pem 2023/02/08 100116 [INFO] [node-controller] Creating CA: /management-state/node/nodes/vafnode2/certs/ca.pem 2023/02/08 100116 [INFO] [node-controller] Creating client certificate: /management-state/node/nodes/vafnode1/certs/cert.pem 2023/02/08 100117 [INFO] [node-controller] Creating client certificate: /management-state/node/nodes/vafnode2/certs/cert.pem 2023/02/08 100117 [INFO] [node-controller] Running pre-create checks... 2023/02/08 100117 [INFO] [node-controller] (vafnode1) Connecting to vSphere for pre-create checks... 2023/02/08 100117 [INFO] [node-controller] Running pre-create checks... 2023/02/08 100117 [INFO] [node-controller] (vafnode2) Connecting to vSphere for pre-create checks... 2023/02/08 100118 [INFO] [node-controller] Creating client certificate: /management-state/node/nodes/vafnode3/certs/cert.pem 2023/02/08 100118 [INFO] [node-controller] Running pre-create checks... 2023/02/08 100118 [INFO] [node-controller] (vafnode3) Connecting to vSphere for pre-create checks... 2023/02/08 100120 [INFO] [node-controller] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2023/02/08 100120 [INFO] [node-controller] 2023/02/08 100120 [INFO] [node-controller] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2023/02/08 100120 [INFO] [node-controller] 2023/02/08 100120 [INFO] [node-controller] Generating and uploading node config vafnode2 2023/02/08 100120 [INFO] [node-controller] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2023/02/08 100120 [INFO] [node-controller] 2023/02/08 100120 [INFO] [node-controller] Generating and uploading node config vafnode1 2023/02/08 100121 [INFO] [node-controller] Wrapper Docker Machine process exiting due to closed plugin server (connection is shut down) 2023/02/08 100121 [INFO] [node-controller] Generating and uploading node config vafnode3 2023/02/08 100135 [ERROR] error syncing 'c-tzmzg/m-4zs56': handler node-controller: Error with pre-create check: "unexpected EOF", requeuing 2023/02/08 100136 [ERROR] error syncing 'c-tzmzg/m-gnkfp': handler node-controller: Error with pre-create check: "unexpected EOF", requeuing 2023/02/08 100136 [ERROR] error syncing 'c-tzmzg/m-6m4wz': handler node-controller: Error with pre-create check: "unexpected EOF", requeuing

agreeable-oil-87482

02/08/2023, 10:06 AM

In the local

Rancher

cluster, look for a

job

in the fleet-default namespace that resembles your node name. You'll get more specific provisioning logs from there

Usually if it doesn't progress past

Connecting to vSphere for pre-create checks...

it's down to routing/firewalls/auth

hundreds-twilight-31144

02/08/2023, 12:02 PM

i can not find any job in fleet-default namespace. root@enes-rancher:~# kubectl get jobs --all-namespaces NAMESPACE NAME COMPLETIONS DURATION AGE kube-system helm-install-traefik 1/1 26s 4d23h kube-system helm-install-traefik-crd 1/1 23s 4d23h Pool: vafnode VMware vSphere – No Location / 2.15 G, 2 Core (template_no_dhcp) 0 of 3 Saving m-4kqxv — - — All 2.2 mins Error with pre-create check: "unexpected EOF":Timeout waiting for ssh key Saving m-p5j6b — - — All 2.2 mins Error with pre-create check: "unexpected EOF":Timeout waiting for ssh key Saving m-z5nlf — - — All 2.2 mins Error with pre-create check: "unexpected EOF":Timeout waiting for ssh key Condition Status Updated Message BackingNamespaceCreated True 2.5 hours ago — Connected False 2.5 hours ago — Created True 2.5 hours ago — CreatorMadeOwner True 2.5 hours ago — DefaultProjectCreated True 2.5 hours ago — InitialRolesPopulated True 2.5 hours ago — NoDiskPressure True 2.5 hours ago — NoMemoryPressure True 2.5 hours ago — Pending True - — Reconciling True 2.5 hours ago [Reconciling] RKECluster True 2.5 hours ago — RKESecretsMigrated True 2.5 hours ago — SecretsMigrated True 2.5 hours ago — ServiceAccountSecretsMigrated True 2.5 hours ago — Stalled False 2.5 hours ago — SystemProjectCreated True 2.5 hours ago — Waiting Unknown 2.5 hours ago Waiting for API to be available

agreeable-oil-87482

02/08/2023, 12:13 PM

It'll be removed after a while, I would try and provision a new cluster and monitor that ns for the job

hundreds-twilight-31144

02/08/2023, 1:03 PM

When i try to create rke2 cluster i see the job, with creation of rke1 cluster noting happen This is the error messages from that process events Active 4m46s Warning BackoffLimitExceeded job/vknode-pool1-aad64d9d-6qcwz-machine-provision job-controller Job has reached the specified backoff limit rancher logs 2023/02/08 130212 [ERROR] error syncing 'fleet-default/vknode-bootstrap-template-g79kh': handler rke-machine: failed to delete fleet-default/vknode-bootstrap-template-g79kh-machine-plan /v1, Kind=Secret for rke-machine fleet-default/vknode-bootstrap-template-g79kh: secrets "vknode-bootstrap-template-g79kh-machine-plan" not found, failed to delete fleet-default/vknode-bootstrap-template-g79kh-machine-bootstrap /v1, Kind=ServiceAccount for rke-machine fleet-default/vknode-bootstrap-template-g79kh: serviceaccounts "vknode-bootstrap-template-g79kh-machine-bootstrap" not found, failed to delete fleet-default/vknode-bootstrap-template-g79kh-machine-plan rbac.authorization.k8s.io/v1, Kind=RoleBinding for rke-machine fleet-default/vknode-bootstrap-template-g79kh: rolebindings.rbac.authorization.k8s.io "vknode-bootstrap-template-g79kh-machine-plan" not found, requeuing 2023/02/08 130212 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130212 [ERROR] [CAPI] Reconciler error: machines.cluster.x-k8s.io "vknode-pool1-85c45798dc-k2np7" not found 2023/02/08 130213 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130213 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [ERROR] error syncing 'fleet-default/vknode-pool1-aad64d9d-2kdhg': handler machine-provision: no machine owner ref, requeuing 2023/02/08 130214 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-p99n6 2023/02/08 130215 [ERROR] error syncing 'local/crtb-9w4q8': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: namespaces "p-p99n6" not found, requeuing 2023/02/08 130215 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130216 [INFO] EnsureSecretForServiceAccount: waiting for secret [vknode-bootstrap-template-8sbhm-machine-bootstrap-token-j9f6v] to be populated with token 2023/02/08 130216 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130216 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130217 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130217 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130217 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130218 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130220 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130236 [INFO] [MachineProvision] Failed to create infrastructure fleet-default/vknode-pool1-aad64d9d-2kdhg for machine vknode-pool1-85c45798dc-vjr7t, deleting and recreating... 2023/02/08 130236 [INFO] [MachineProvision] Failed to create infrastructure fleet-default/vknode-pool1-aad64d9d-2kdhg for machine vknode-pool1-85c45798dc-vjr7t, deleting and recreating... 2023/02/08 130236 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130236 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130236 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130237 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130237 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130237 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130238 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130244 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130245 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130245 [ERROR] error syncing 'fleet-default/vknode-bootstrap-template-8sbhm': handler rke-machine: failed to delete fleet-default/vknode-bootstrap-template-8sbhm-machine-plan /v1, Kind=Secret for rke-machine fleet-default/vknode-bootstrap-template-8sbhm: secrets "vknode-bootstrap-template-8sbhm-machine-plan" not found, failed to delete fleet-default/vknode-bootstrap-template-8sbhm-machine-bootstrap /v1, Kind=ServiceAccount for rke-machine fleet-default/vknode-bootstrap-template-8sbhm: serviceaccounts "vknode-bootstrap-template-8sbhm-machine-bootstrap" not found, failed to delete fleet-default/vknode-bootstrap-template-8sbhm-machine-plan /v1, Kind=ServiceAccount for rke-machine fleet-default/vknode-bootstrap-template-8sbhm: serviceaccounts "vknode-bootstrap-template-8sbhm-machine-plan" not found, requeuing 2023/02/08 130245 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130246 [ERROR] [CAPI] Reconciler error: machines.cluster.x-k8s.io "vknode-pool1-85c45798dc-vjr7t" not found 2023/02/08 130246 [ERROR] error syncing 'fleet-default/vknode-pool1-aad64d9d-nskmb': handler machine-provision: no machine owner ref, requeuing 2023/02/08 130246 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130246 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130246 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130247 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130248 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130248 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node

just to add Condition Status Updated Message BackingNamespaceCreated True 22 mins ago — Connected False 22 mins ago — Created True 31 secs ago — CreatorMadeOwner True 22 mins ago — DefaultProjectCreated True 22 mins ago — InitialRolesPopulated True 22 mins ago — NoDiskPressure True 22 mins ago — NoMemoryPressure True 22 mins ago — Provisioned True 21 mins ago — Ready False 21 mins ago [Disconnected] Cluster agent is not connected Reconciling True 22 mins ago [Reconciling] RKECluster True 21 mins ago — RKESecretsMigrated True 22 mins ago — SecretsMigrated True 22 mins ago — ServiceAccountSecretsMigrated True 22 mins ago — Stalled False 22 mins ago — SystemProjectCreated True 22 mins ago — Updated Unknown 21 mins ago [Waiting] waiting for viable init node

agreeable-oil-87482

02/08/2023, 1:30 PM

What are the logs from vknode-pool1-aad64d9d-6qcwz-machine-provision?

hundreds-twilight-31144

02/08/2023, 1:35 PM

Hello, i create another cluster the same error Running pre-create checks... (vk8node-pool1-9412b1b0-qbd7p) Connecting to vSphere for pre-create checks... Error with pre-create check: "unexpected EOF" The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.

agreeable-oil-87482

02/08/2023, 1:37 PM

I'd check the VPXD logs next, looks like your vcenter server is returning something invalid

hundreds-twilight-31144

02/08/2023, 3:22 PM

Hello, i inspect logs and didnt find anything strange.. just to mention that network in vcentar is using distributed switch and we have two clusters separated..

I investigate rancher-machine and found some errors rancher-machine --debug create -d vmwarevsphere --engine-install-url https://releases.rancher.com/install-docker/20.10.sh \ --vmwarevsphere-clone-from "xxx" \ --vmwarevsphere-ssh-user docker \ --vmwarevsphere-datastore "xxx" \ --vmwarevsphere-os linux \ --vmwarevsphere-vcenter-port 443 \ --vmwarevsphere-datacenter "xxx" \ --vmwarevsphere-memory-size 2048 \ --vmwarevsphere-ssh-port 22 \ --vmwarevsphere-ssh-password tcuser \ --vmwarevsphere-cfgparam disk.enableUUID=TRUE \ --vmwarevsphere-password xxx \ --vmwarevsphere-pool "/DR Data Centar/host/B200_M4/Resources/Rancher_Pool" \ --vmwarevsphere-ssh-user-group staff \ --vmwarevsphere-creation-type template \ --vmwarevsphere-disk-size 20000 \ --vmwarevsphere-username xxx \ --vmwarevsphere-vcenter xxx \ --vmwarevsphere-hostsystem "xxx" \ --vmwarevsphere-network "/DR Data Centar/network/DR_vDS_01-RKE_CLUSTER-VLAN_3495" \ --vmwarevsphere-cpu-count 2 vk8node2 rancher gui send this with network flag /DR Data Centar/network/DR_vDS_01-RKE_CLUSTER-VLAN_3495 but when i manual change it and executed it on rancher-machine to DR_vDS_01-RKE_CLUSTER-VLAN_3495 it creates vm on vcentar. We are using distributed port group...do you have any suggestions thank you

We found solution for this problem. We have to edit Nodetemplate throug cli and set correct network path.

agreeable-oil-87482

02/09/2023, 9:29 PM

Interesting. You shouldn't need to, vds should work out of the box

But glad you found a workaround

hundreds-twilight-31144

02/10/2023, 2:29 PM

Thank you very much David

211 Views