I guess something between the LB and the VMs isn't working

Maybe something else that's relevant - I recently had to delete and re-add my second node, using the same name and ip address and all that

And accessing the LB for the Harvester UI works fine

Ah, you gave a good hint. The VMs cannot ping the harvester nodes (.60,.61,.62)

The VMs can ping each other though

so basically you have to have a host network and VLAN that's fully addressable from the mgmt network.

Are you running the VMs off the mgmt network or some other network?

If I turn off all NICs on the server except the "mgmt" one (10.0.1.61/62) then the ping from host to VM still works

traceroute to 10.5.106.77 (10.5.106.77), 30 hops max, 60 byte packets
 1  10.0.1.32 (10.0.1.32)  0.689 ms  0.622 ms  0.590 ms
 2  10.5.106.77 (10.5.106.77)  0.565 ms  0.539 ms  0.513 ms

This is from a host to a VM - passes through the gateway

opensuse@test:~> ip route show
default via 10.5.106.254 dev eth0 
10.5.106.0/24 dev eth0 proto kernel scope link src 10.5.106.77

That's the correct gateway for the VM to reach the servers too But a traceroute to a host doesn't even pass through there

opensuse@test:~> sudo traceroute 10.0.1.61  
traceroute to 10.0.1.61 (10.0.1.61), 30 hops max, 60 byte packets
 1  * * *
 2  * * *

Created a test VM - for network it's only on mgmt in bridge. That one works to ping a host. So I guess I have to add the other VMs to the mgmt network as well, with that network in bridge

Well my VMs won't take IPs for both mgmt and my own vlan. Either way, this has only recently become a case. My LB has been load balancing for months so far and now they just don't.

Harvester UI says that there are 2 IPs on the VM. Good. In the VM, I have 2 interfaces, eth0 and eth1. Good. eth0 is down by default. Bad. eth1 has the VLAN IP. VM still cannot reach hosts. Host cannot ping the mgmt IP. Loadbalancer still times out.

Worth mentioning! Harvester 1 has the following routes:

Harvester 2, which got deleted and subsequently re-added, has only the following:

This is the svc for my lb.

harvester-1:/home/rancher # kubectl describe svc rancher-lb -n rancher-mgmt
Name:                     rancher-lb
Namespace:                rancher-mgmt
Labels:                   <http://loadbalancer.harvesterhci.io/servicelb=true|loadbalancer.harvesterhci.io/servicelb=true>
Annotations:              <http://kube-vip.io/ignore-service-security|kube-vip.io/ignore-service-security>: true
                          <http://kube-vip.io/loadbalancerIPs|kube-vip.io/loadbalancerIPs>: 10.0.1.70
                          <http://kube-vip.io/vipHost|kube-vip.io/vipHost>: harvester-1
Selector:                 <none>
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.53.1.100
IPs:                      10.53.1.100
IP:                       10.0.1.70
LoadBalancer Ingress:     10.0.1.70
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32390/TCP
Endpoints:                <none>
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  32047/TCP
Endpoints:                <none>
Port:                     kubeapi  6443/TCP
TargetPort:               6443/TCP
NodePort:                 kubeapi  32108/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Things here which are missing from, say, the ingress-expose svc: • selector is none • there are two values for IP? • No endpoint values. For the ingress-expose, the service's endpoints are the pod IPs for the nginx ingress