https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# vsphere
a
This message was deleted.
h
I try David repo all ubuntu variations with packer, downloading cloud img. We stuck on pre create checks and dont see that vcentar is generating vm from template that we prepare. Credentials are ok. Vcentar and rancher are in different vlans
Just to mention that rancher is installed on single node using helm
a
So you see the user logged into vcenter that you're using in your cloud credential in Rancher but no vm cloning activity? • Have you checked the vcenter server vpxd logs for that user? • Have you tried logging into vcenter with the these user credentials to verify it has access to the vm templates etc?
h
in rancher logs
2023/02/08 100116 [INFO] [node-controller] Creating CA: /management-state/node/nodes/vafnode1/certs/ca.pem 2023/02/08 100116 [INFO] [node-controller] Creating CA: /management-state/node/nodes/vafnode3/certs/ca.pem 2023/02/08 100116 [INFO] [node-controller] Creating CA: /management-state/node/nodes/vafnode2/certs/ca.pem 2023/02/08 100116 [INFO] [node-controller] Creating client certificate: /management-state/node/nodes/vafnode1/certs/cert.pem 2023/02/08 100117 [INFO] [node-controller] Creating client certificate: /management-state/node/nodes/vafnode2/certs/cert.pem 2023/02/08 100117 [INFO] [node-controller] Running pre-create checks... 2023/02/08 100117 [INFO] [node-controller] (vafnode1) Connecting to vSphere for pre-create checks... 2023/02/08 100117 [INFO] [node-controller] Running pre-create checks... 2023/02/08 100117 [INFO] [node-controller] (vafnode2) Connecting to vSphere for pre-create checks... 2023/02/08 100118 [INFO] [node-controller] Creating client certificate: /management-state/node/nodes/vafnode3/certs/cert.pem 2023/02/08 100118 [INFO] [node-controller] Running pre-create checks... 2023/02/08 100118 [INFO] [node-controller] (vafnode3) Connecting to vSphere for pre-create checks... 2023/02/08 100120 [INFO] [node-controller] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2023/02/08 100120 [INFO] [node-controller] 2023/02/08 100120 [INFO] [node-controller] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2023/02/08 100120 [INFO] [node-controller] 2023/02/08 100120 [INFO] [node-controller] Generating and uploading node config vafnode2 2023/02/08 100120 [INFO] [node-controller] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag. 2023/02/08 100120 [INFO] [node-controller] 2023/02/08 100120 [INFO] [node-controller] Generating and uploading node config vafnode1 2023/02/08 100121 [INFO] [node-controller] Wrapper Docker Machine process exiting due to closed plugin server (connection is shut down) 2023/02/08 100121 [INFO] [node-controller] Generating and uploading node config vafnode3 2023/02/08 100135 [ERROR] error syncing 'c-tzmzg/m-4zs56': handler node-controller: Error with pre-create check: "unexpected EOF", requeuing 2023/02/08 100136 [ERROR] error syncing 'c-tzmzg/m-gnkfp': handler node-controller: Error with pre-create check: "unexpected EOF", requeuing 2023/02/08 100136 [ERROR] error syncing 'c-tzmzg/m-6m4wz': handler node-controller: Error with pre-create check: "unexpected EOF", requeuing
a
In the local 
Rancher
cluster, look for a 
job
in the fleet-default namespace that resembles your node name. You'll get more specific provisioning logs from there
Usually if it doesn't progress past 
Connecting to vSphere for pre-create checks...
it's down to routing/firewalls/auth
h
i can not find any job in fleet-default namespace. root@enes-rancher:~# kubectl get jobs --all-namespaces NAMESPACE NAME COMPLETIONS DURATION AGE kube-system helm-install-traefik 1/1 26s 4d23h kube-system helm-install-traefik-crd 1/1 23s 4d23h Pool: vafnode VMware vSphere – No Location / 2.15 G, 2 Core (template_no_dhcp) 0 of 3 Saving m-4kqxv — - — All 2.2 mins Error with pre-create check: "unexpected EOF":Timeout waiting for ssh key Saving m-p5j6b — - — All 2.2 mins Error with pre-create check: "unexpected EOF":Timeout waiting for ssh key Saving m-z5nlf — - — All 2.2 mins Error with pre-create check: "unexpected EOF":Timeout waiting for ssh key Condition Status Updated Message BackingNamespaceCreated True 2.5 hours ago — Connected False 2.5 hours ago — Created True 2.5 hours ago — CreatorMadeOwner True 2.5 hours ago — DefaultProjectCreated True 2.5 hours ago — InitialRolesPopulated True 2.5 hours ago — NoDiskPressure True 2.5 hours ago — NoMemoryPressure True 2.5 hours ago — Pending True - — Reconciling True 2.5 hours ago [Reconciling] RKECluster True 2.5 hours ago — RKESecretsMigrated True 2.5 hours ago — SecretsMigrated True 2.5 hours ago — ServiceAccountSecretsMigrated True 2.5 hours ago — Stalled False 2.5 hours ago — SystemProjectCreated True 2.5 hours ago — Waiting Unknown 2.5 hours ago Waiting for API to be available
a
It'll be removed after a while, I would try and provision a new cluster and monitor that ns for the job
h
When i try to create rke2 cluster i see the job, with creation of rke1 cluster noting happen This is the error messages from that process events Active 4m46s Warning BackoffLimitExceeded job/vknode-pool1-aad64d9d-6qcwz-machine-provision job-controller Job has reached the specified backoff limit rancher logs 2023/02/08 130212 [ERROR] error syncing 'fleet-default/vknode-bootstrap-template-g79kh': handler rke-machine: failed to delete fleet-default/vknode-bootstrap-template-g79kh-machine-plan /v1, Kind=Secret for rke-machine fleet-default/vknode-bootstrap-template-g79kh: secrets "vknode-bootstrap-template-g79kh-machine-plan" not found, failed to delete fleet-default/vknode-bootstrap-template-g79kh-machine-bootstrap /v1, Kind=ServiceAccount for rke-machine fleet-default/vknode-bootstrap-template-g79kh: serviceaccounts "vknode-bootstrap-template-g79kh-machine-bootstrap" not found, failed to delete fleet-default/vknode-bootstrap-template-g79kh-machine-plan rbac.authorization.k8s.io/v1, Kind=RoleBinding for rke-machine fleet-default/vknode-bootstrap-template-g79kh: rolebindings.rbac.authorization.k8s.io "vknode-bootstrap-template-g79kh-machine-plan" not found, requeuing 2023/02/08 130212 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130212 [ERROR] [CAPI] Reconciler error: machines.cluster.x-k8s.io "vknode-pool1-85c45798dc-k2np7" not found 2023/02/08 130213 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130213 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [ERROR] error syncing 'fleet-default/vknode-pool1-aad64d9d-2kdhg': handler machine-provision: no machine owner ref, requeuing 2023/02/08 130214 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130214 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-p99n6 2023/02/08 130215 [ERROR] error syncing 'local/crtb-9w4q8': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: namespaces "p-p99n6" not found, requeuing 2023/02/08 130215 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130216 [INFO] EnsureSecretForServiceAccount: waiting for secret [vknode-bootstrap-template-8sbhm-machine-bootstrap-token-j9f6v] to be populated with token 2023/02/08 130216 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130216 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130217 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130217 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130217 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130218 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130220 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130236 [INFO] [MachineProvision] Failed to create infrastructure fleet-default/vknode-pool1-aad64d9d-2kdhg for machine vknode-pool1-85c45798dc-vjr7t, deleting and recreating... 2023/02/08 130236 [INFO] [MachineProvision] Failed to create infrastructure fleet-default/vknode-pool1-aad64d9d-2kdhg for machine vknode-pool1-85c45798dc-vjr7t, deleting and recreating... 2023/02/08 130236 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130236 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130236 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130237 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130237 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130237 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130238 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130244 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130245 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130245 [ERROR] error syncing 'fleet-default/vknode-bootstrap-template-8sbhm': handler rke-machine: failed to delete fleet-default/vknode-bootstrap-template-8sbhm-machine-plan /v1, Kind=Secret for rke-machine fleet-default/vknode-bootstrap-template-8sbhm: secrets "vknode-bootstrap-template-8sbhm-machine-plan" not found, failed to delete fleet-default/vknode-bootstrap-template-8sbhm-machine-bootstrap /v1, Kind=ServiceAccount for rke-machine fleet-default/vknode-bootstrap-template-8sbhm: serviceaccounts "vknode-bootstrap-template-8sbhm-machine-bootstrap" not found, failed to delete fleet-default/vknode-bootstrap-template-8sbhm-machine-plan /v1, Kind=ServiceAccount for rke-machine fleet-default/vknode-bootstrap-template-8sbhm: serviceaccounts "vknode-bootstrap-template-8sbhm-machine-plan" not found, requeuing 2023/02/08 130245 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130246 [ERROR] [CAPI] Reconciler error: machines.cluster.x-k8s.io "vknode-pool1-85c45798dc-vjr7t" not found 2023/02/08 130246 [ERROR] error syncing 'fleet-default/vknode-pool1-aad64d9d-nskmb': handler machine-provision: no machine owner ref, requeuing 2023/02/08 130246 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130246 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130246 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130247 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130248 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node 2023/02/08 130248 [INFO] [planner] rkecluster fleet-default/vknode: waiting: waiting for viable init node
just to add Condition Status Updated Message BackingNamespaceCreated True 22 mins ago — Connected False 22 mins ago — Created True 31 secs ago — CreatorMadeOwner True 22 mins ago — DefaultProjectCreated True 22 mins ago — InitialRolesPopulated True 22 mins ago — NoDiskPressure True 22 mins ago — NoMemoryPressure True 22 mins ago — Provisioned True 21 mins ago — Ready False 21 mins ago [Disconnected] Cluster agent is not connected Reconciling True 22 mins ago [Reconciling] RKECluster True 21 mins ago — RKESecretsMigrated True 22 mins ago — SecretsMigrated True 22 mins ago — ServiceAccountSecretsMigrated True 22 mins ago — Stalled False 22 mins ago — SystemProjectCreated True 22 mins ago — Updated Unknown 21 mins ago [Waiting] waiting for viable init node
a
What are the logs from vknode-pool1-aad64d9d-6qcwz-machine-provision?
h
Hello, i create another cluster the same error Running pre-create checks... (vk8node-pool1-9412b1b0-qbd7p) Connecting to vSphere for pre-create checks... Error with pre-create check: "unexpected EOF" The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
a
I'd check the VPXD logs next, looks like your vcenter server is returning something invalid
h
Hello, i inspect logs and didnt find anything strange.. just to mention that network in vcentar is using distributed switch and we have two clusters separated..
I investigate rancher-machine and found some errors rancher-machine --debug create -d vmwarevsphere --engine-install-url https://releases.rancher.com/install-docker/20.10.sh \ --vmwarevsphere-clone-from "xxx" \ --vmwarevsphere-ssh-user docker \ --vmwarevsphere-datastore "xxx" \ --vmwarevsphere-os linux \ --vmwarevsphere-vcenter-port 443 \ --vmwarevsphere-datacenter "xxx" \ --vmwarevsphere-memory-size 2048 \ --vmwarevsphere-ssh-port 22 \ --vmwarevsphere-ssh-password tcuser \ --vmwarevsphere-cfgparam disk.enableUUID=TRUE \ --vmwarevsphere-password xxx \ --vmwarevsphere-pool "/DR Data Centar/host/B200_M4/Resources/Rancher_Pool" \ --vmwarevsphere-ssh-user-group staff \ --vmwarevsphere-creation-type template \ --vmwarevsphere-disk-size 20000 \ --vmwarevsphere-username xxx \ --vmwarevsphere-vcenter xxx \ --vmwarevsphere-hostsystem "xxx" \ --vmwarevsphere-network "/DR Data Centar/network/DR_vDS_01-RKE_CLUSTER-VLAN_3495" \ --vmwarevsphere-cpu-count 2 vk8node2 rancher gui send this with network flag /DR Data Centar/network/DR_vDS_01-RKE_CLUSTER-VLAN_3495 but when i manual change it and executed it on rancher-machine to DR_vDS_01-RKE_CLUSTER-VLAN_3495 it creates vm on vcentar. We are using distributed port group...do you have any suggestions thank you
We found solution for this problem. We have to edit Nodetemplate throug cli and set correct network path.
a
Interesting. You shouldn't need to, vds should work out of the box
But glad you found a workaround
h
Thank you very much David
311 Views
Open in Slack
PreviousNext
Powered by