https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
f
What's happening: the LB's backend membership comes from 
vmi.status.interfaces[].infoSource
containing 
guest-agent
. That intermittently disappears for the backing VMIs — sometimes one at a time, sometimes all three simultaneously — and the LB drops to 
no running backend servers
whenever it does. It self-recovers, then repeats. I confirmed the guest side is not at fault: querying the agent directly the whole time it's flapping —
Copy code
$ virsh qemu-agent-command <domain> '{"execute":"guest-network-get-interfaces"}'
— returns correct, completely unchanged data (same IP, same interface count) continuously, even during the exact window where 
vmi.status.interfaces
has already dropped it. `virt-launcher`'s own agent-poller logs show it successfully polling the agent every 5-10s the entire time, no errors. So the agent connection is fine; something between `virt-launcher`'s cached data and the VMI status write is dropping it.
This looks like the same class of issue as kubevirt/kubevirt#12698 and harvester/harvester#7031. I tried the workaround from #7031 (setting 
<http://harvesterhci.io/waitForLeaseInterfaceNames|harvesterhci.io/waitForLeaseInterfaceNames>
to match the guest's real interface name, 
enp1s0
, then restarting the VM) — it bought a longer stable window right after restart (~2.5 min vs the usual cycle) but the flapping resumed. Also tried restarting 
virt-handler
on the affected node — briefly cleared it, but a VM on a different, non-restarted node dropped out on the same cycle shortly after, so it doesn't look like a stuck process. One thing I can't explain: other VMs on the same Harvester install — including one with no 
waitForLeaseInterfaceNames
annotation at all and 20 guest interfaces (vs. 6 on the affected VMs) — are completely stable. Only the Rancher-node-driver-provisioned guest-cluster VMs are affected, everything else on the cluster is fine. Not sure if that's a real clue or coincidence.
I use the harvester load balancer feature just fine to run Rancher's dashboard without issue. Also on VM with RKE2.
a
f
Thanks @abundant-account-48953! I spent some time on this and think it is a bug with kubevirt. I worked around it by moving the K8S VMs off their own VLAN and onto the same VLAN harvester runs on. Seems stable that way- but was a frustrating issue to find and troubleshoot! > • This matches `kubevirt/kubevirt#12698`more closely than`#9517`— same symptom class (guest-agent-sourced interface data disappearing despite the agent staying connected), open/unresolved since 2024, no confirmed root cause upstream either. > • The one variable that actually correlated with the bug in our environment: VMs on a custom VLAN-tagged multus network (
net-vlan1056
) hit it consistently; VMs on the plain default/untagged network never did, across every comparison we ran (including a long-lived control-plane node and, eventually, 3 freshly-migrated production nodes). Never fully pinned the internal mechanism for_why_— that'd need KubeVirt source/debug-level access we didn't have.
> Fix applied: migrated the affected guest cluster's node pool off the VLAN-tagged network onto the plain default network. LB confirmed stable for 10+ minutes post-migration (zero flaps, vs. the old cyclic pattern) — first time it's held steady since this started.
3 Views
Open in Slack
PreviousNext
Powered by