https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
h
Maybe @witty-jelly-95845 has insight? I have looked at the CRD description and I think I have everything correct. My next troubleshooting step was to use Postman to test the vCenter SDK to confirm it is working properly. I successfully browse to http://vcenterDomain/sdk/vimService.wsdl Thank you Thank you!
g
hi, do you see anything in the import controller logs?
does the endpoint contain 
sdk
suffix?
Copy code
"<https://vscim/sdk>"
h
Thank you @great-bear-19718, looking now ...
g
we use govc.. and it needs 
/sdk
as suffix to vcenter url
h
yes, my endpoint does contain the /sdk suffix - here is the vcsim-crd.yaml I applied:
Copy code
apiVersion: <http://migration.harvesterhci.io/v1beta1|migration.harvesterhci.io/v1beta1>
kind: VmwareSource
metadata:
  name: vcsim
  namespace: default
spec:
  endpoint: "<http://vcenterdomainhere/sdk>"
  dc: "GW"
  credentials:
    name: vsphere-credentials
    namespace: default
g
great.. then we just need to see what is going on the import-controller logs..
it basically takes the creds and checks if it can list the objects in datacenter..
h
where do I find the import-controller logs?
g
i will have to check the code but i forgot now
kubectl get pods -n harvester-system
you should see a pod for 
harvester-vm-import-controller-
there
h
got it! just a moment as I pull the logs ...
Copy code
fmh1n1:~ # kubectl get pods -n harvester-system | grep import
vm-import-controller-harvester-vm-import-controller-8588878lkz2   1/1     Running   0             8d
now I remember, I tried this before
Copy code
fmh1n1:~ # kubectl get log vm-import-controller-harvester-vm-import-controller-8588878lkz2
error: the server doesn't have a resource type "log"
g
kubectl logs vm-import-controller-harvester-vm-import-controller-8588878lkz2 -n harvester-system
h
my bad, sorry, just a moment - thank you ...
Copy code
time="2023-02-06T17:04:32Z" level=error msg="error syncing 'default/vcenter.domainhere': handler vmware-migration-change: error generating vmware client for vmware migration: vcenter.domainhere: error creating vim client: Post \"<https://vcenter.domainhere/>\": dial tcp 192.168.10.49:443: i/o timeout, requeuing"
time="2023-02-06T17:09:32Z" level=info msg="reoncilling vmware migration default/vcenter.domainhere"


time="2023-02-06T19:34:17Z" level=error msg="error syncing 'default/vcsim': handler vmware-migration-change: error generating vmware client for vmware migration: vcsim: error creating vim client: Post \"<http://vcenterIPv4here/sdk>\": dial tcp vcenterIPv4here:80: i/o timeout, requeuing"
g
it cant get to either endpoint..
h
also
Copy code
time="2023-02-09T22:53:39Z" level=info msg="reoncilling vmware migration default/vcsim"
Thank you, I am looking at it ...
is there a way to see the headers and body for this:
Copy code
Post \"<http://vcenter.domain/sdk>\"
the subsequent IP in
Copy code
dial tcp <vcenterIPhere>:80
^ is correct
here is the return of that Post from Postman:
Copy code
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="<http://schemas.xmlsoap.org/soap/encoding/>"
 xmlns:soapenv="<http://schemas.xmlsoap.org/soap/envelope/>"
 xmlns:xsd="<http://www.w3.org/2001/XMLSchema>"
 xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>">
    <soapenv:Body>
        <soapenv:Fault>
            <faultcode>ClientFaultCode</faultcode>
            <faultstring>
Error returned by expat parser: no element found

while parsing HTTP request before method was determined
at line 1, column 0</faultstring>
        </soapenv:Fault>
    </soapenv:Body>
</soapenv:Envelope>
^ obviously this was a malformed SOAP post on my part ...
g
does vcenter allow 
http
endpoints?
h
I do not have a valid certificate at
Copy code
time="2023-02-06T17:38:06Z" level=error msg="error syncing 'default/vcenter.domain': handler vmware-migration-change: error generating vmware client for vmware migration: vcenter.domain: error creating vim client: Post \"<https://vcenter.domain/sdk>\": dial tcp vcenterIPv4:443: i/o timeout, requeuing"
I just saw that it is trying it at 443
I thought I enabled http, just a minute while I recheck
g
should be fine if its not a valid cert..
if there is no cert supplied in the secret then it will use tls ignore option
does your harvester host have multiple nics?
h
Great to here, the secret applied is
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: vsphere-credentials
  namespace: default
stringData:
  "username": "administrator@vsphere.domain"
  "password": "passwordhere"
yes, 3x Harvester nodes, each with 8 NICS: • 2x 10G LACP bond for management network • 2x 10G LACP bond for vm networks • 2x 10G LACP bond for storage • 1x 1G to out-of-band network • 1x 1G empty
g
is vcenter accessible over management network nics?
because that is what k8s overlay will be using for pod traffic egress
h
I did go through the VMware docs to enable http at /sdk
g
its possible you are able to ping vcenter from the host, but ping is not using the management interfaces
h
good point! testing
does not appear to ping from mgmt-bo
Copy code
fmh1n1:~ # ping -c 3 vcenterIP
PING vcenterIP (vcenterIP) 56(84) bytes of data.
64 bytes from 192.168.10.49: icmp_seq=1 ttl=62 time=0.589 ms
64 bytes from 192.168.10.49: icmp_seq=2 ttl=62 time=0.472 ms
64 bytes from 192.168.10.49: icmp_seq=3 ttl=62 time=0.602 ms

--- vcenterIP ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2051ms
rtt min/avg/max/mdev = 0.472/0.554/0.602/0.061 ms

fmh1n1:~ # ping -c 3 vcenter.domain
PING vcenter.domain (vcenterIP) 56(84) bytes of data.
64 bytes from vcenterIP: icmp_seq=1 ttl=62 time=0.666 ms
64 bytes from vcenterIP: icmp_seq=2 ttl=62 time=0.482 ms
64 bytes from vcenterIP: icmp_seq=3 ttl=62 time=0.477 ms

--- vcenter.domain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2018ms
rtt min/avg/max/mdev = 0.477/0.541/0.666/0.091 ms
fmh1n1:~ #


fmh1n1:~ # ping -c 3 -I mgmt-bo vcenter.domain
ping: Warning: source address might be selected on device other than mgmt-bo.
PING vcenter.domain (vcenterIP) from 172.16.12.21 mgmt-bo: 56(84) bytes of data.

--- vcenter.domain ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2055ms
g
that would likely be the reason
h
THANK YOU @great-bear-19718 - incredibly helpful - I thought I had it covered with the original pings but forgot to make sure they were routing out the correct interface
🙏
Hi @great-bear-19718, I formed my ping wrong - the interface is 'mgmt-br' not 'mgmt-bo' as I had tried:
Copy code
fmh1n1:~ # ping -c 3 -I mgmt-br vcenter.domain
PING vcenter.domain (vcenterIP) from 172.16.12.21 mgmt-br: 56(84) bytes of data.
64 bytes from vcenterIP: icmp_seq=1 ttl=62 time=0.516 ms
64 bytes from vcenterIP: icmp_seq=2 ttl=62 time=0.457 ms
64 bytes from vcenterIP: icmp_seq=3 ttl=62 time=0.404 ms

--- vcenter.domain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.404/0.459/0.516/0.045 ms
Copy code
ip link
...
10: mgmt-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:c3:b5:16 brd ff:ff:ff:ff:ff:ff


11: mgmt-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master mgmt-br state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:c3:b5:16 brd ff:ff:ff:ff:ff:ff
...
104: cali656d3dc46a8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-95ab663e-f2e5-0ce4-d7c8-cf23696c6cf8


128: sn-br: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether ce:d1:bc:fa:8a:ff brd ff:ff:ff:ff:ff:ff
132: ifn-br: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether a2:81:7a:b4:63:c4 brd ff:ff:ff:ff:ff:ff


135: cali2efafb2c278@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-89a29d2f-451d-d990-6c4b-77eaba9d5125
136: cali9cf3d080d5b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-b209fad5-a57f-d6c7-09ea-407264791ad0
137: veth3976ab13@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master sn-br state UP mode DEFAULT group default
    link/ether 22:e0:bc:27:e4:6a brd ff:ff:ff:ff:ff:ff link-netns cni-89a29d2f-451d-d990-6c4b-77eaba9d5125
138: veth98bd93f6@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master sn-br state UP mode DEFAULT group default
    link/ether 72:82:99:f2:50:63 brd ff:ff:ff:ff:ff:ff link-netns cni-b209fad5-a57f-d6c7-09ea-407264791ad0

    
183: sn-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master sn-br state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:c3:b5:27 brd ff:ff:ff:ff:ff:ff
184: ifn-bo: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master ifn-br state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:c3:b5:17 brd ff:ff:ff:ff:ff:ff
mgmt is Management network, sn is storage, ifn is tagged vm network
vCenter is configured to allow http:
Copy code
root@vcenter [ /etc/vmware-rhttpproxy/endpoints.conf.d ]# cat vpxd-rhttpproxy-endpoint.conf
/sdk-vpxd local 8085 allow allow
/qs namedpipe /var/run/vmware/vpxd-qsadapter-pipe redirect allow
/vod namedpipe /var/run/vmware/vpxd-webserver-pipe redirect allow
/screen namedpipe /var/run/vmware/vpxd-webserver-pipe redirect allow
/mob namedpipe /var/run/vmware/vpxd-mob-pipe allow allow
vcenter.dtcorp.diamondtours.corp:8089 localtunnel 8089 allow reject
/nfc namedpipe /var/run/vmware/vpxd-webserver-pipe allow allow
/sdkTunnel localtunnel 8089 allow reject
sdkTunnel:8089 localtunnel 8089 allow reject
/vcenter local 8093 redirect allow
/sdk local 8085 allow allow
/client/clients.xml namedpipe /var/run/vmware/vpxd-webserver-pipe allow allow
/catalog namedpipe /var/run/vmware/vpxd-webserver-pipe redirect allow
root@vcenter [ /etc/vmware-rhttpproxy/endpoints.conf.d ]#
note line 
/sdk local 8085 allow allow
is there a way to look at what POST (headers and body) Harvester is sending in
Copy code
time="2023-02-06T17:04:32Z" level=error msg="error syncing 'default/vcenter.domainhere': handler vmware-migration-change: error generating vmware client for vmware migration: vcenter.domainhere: error creating vim client: Post \"<https://vcenter.domainhere/>\": dial tcp vcenterIP:443: i/o timeout, requeuing"
g
it is using 
govc
they have a cli you could try.. it leverages the same stuff
sorry vc is the cli.. you could use that
we leverage the sdk.. https://github.com/vmware/govmomi
h
Thank you! I will look at that tomorrow.
🙏
r
Having this same issue. It seems to be a case of TLS in my case. Getting this error with govc 
govc: Post "<https://100.80.243.158/sdk>": x509: cannot validate certificate for 100.80.243.158 because it doesn't contain any IP SANs
163 Views
Open in Slack
PreviousNext
Powered by