https://rancher.com/ logo
Join Slack
Powered by
We upgraded a test cluster today from Havester 1.5...
# harvester
b
We upgraded a test cluster today from Havester 1.5.1 to 1.6.1. In the middle of the upgrade all life migrations stopped after half the nodes. The reason was, that Harvester 1.6.1 is not using the CPU feature 
<http://cpu-feature.node.kubevirt.io/ipred-ctrl=true|cpu-feature.node.kubevirt.io/ipred-ctrl=true>
by default anymore, but 1.5.1 did. This results in "incompatible" Harvester nodes, the label is not set anymore. As a workaround we set the label on all the nodes to successfully continue with the Harvester upgrade. But Harvesters feature detection is removing the label after a few seconds again. Now the Harvester upgrade is done, but all the VMs cannot be lifemigrated anymore because they still have the feature set in their Nodeselectors. The only way out seems a VM restart (like stop and start to get new VM pods)... I tested this, and it's working. But this is no option for any productive cluster. Has anybody an idea on how to prevent this before or while the Harvester upgrade is running?
b
Seems likely this is related to your CPU model.
I think that label is applied by the kubevirt stack and not configured by Harvester's overlay.
If the kernel upgrade or changes in kubevirt are disabling that feature, there's not gonna be a good way around it.
b
The nodes are all the same, and the CPU model did not change while I upgraded the cluster.
b
Outside of maybe specifying in the defaults to not use that feature to begin with.
This label is associated with the control of the Indirect Predictor (IPRED) behavior, which is related to speculative execution side-channel vulnerabilities:
b
This is a default installation. I don't even know where to configure these things...
b
The linux kernel and/or kubevirt might have changes that alter what features are available.
because updates include security and bug fixes.
b
Of course they have. The flag was removed, so what story does that tell? I'm using the official upgrade path. I don't expect those kind of traps.
BTW: It's not the kernel, I checked that.
b
The story I'd tell is: • Intel screwed up. The "Indirect Predictor" feature isn't safe to run on your model anymore. • kubevirt changed the defaults for your CPU model because reasons. (see above) • You have to change the kubevirt defaults to not use that feature by default. • You already found the way to do that before. Alternatively you can make those changes and reboot the VMs to pick up the new definitions before hand.
I think the kubevirt defaults are something like 
kubectl edit <http://kubevirts.kubevirt.io|kubevirts.kubevirt.io> -n harvester-system
But changes there would still require an instance restart, so it's kinda the same result. There's not really going to be a way around a no reboot afaik.
It'd be a great question if you have paid support though.
b
They are, yes. And there is even a migration configuration to relax the destination check before a life migration. It's not helping for existing pods/VMs.
And yes, I already raised an angry support case. I just thought that somebody from the community went the same way down the rabbit hole and found a way out 😁
b
I don't think my AMD procs have that feature available so it would never hit me.
It's probably related to your specific CPU model.
Been hit by similar issues with a specific CPU model, but edge cases are our job security.
😁 1
b
It's Intel Xeon... but I would not bet that you are save with AMD. This can happen with any of the many features flags.
b
It's might be a VERY SPECIFIC xeon model that this affects.
Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz
isn't the same as 
Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
which is still different from 
Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz
b
Maybe I just found (again) the unicorn. Just like we did some weeks ago, where we had a UUID collision in Longhorn. I'll check tomorrow, but I'd bet we have couple of those boxes.
g
a support bundle would be nice
b
Are you able to access my support case? The number is removed.
g
i can someone to send it in that case
❓ 1
i meant i do not have access directly but i can still get it
👍 1
c
the 
<http://cpu-feature.node.kubevirt.io|cpu-feature.node.kubevirt.io>
labels all come directly from CPU features reported by the underlying hardware. A kernel update and/or microcode update removed this from the feature set for your CPUs, for the reasons @bland-article-62755 mentioned. You cannot simply add them back again. https://github.com/kubevirt/kubevirt/blob/abdfdb88efff132a65190c92dfffc630f4cab68f/pkg/virt-handler/node-labeller/node_labeller.go#L242-L245
I believe you can look at 
/var/lib/kubevirt-node-labeller/supported_features.xml
to confirm that?
👍 1
content comes from here: https://github.com/kubevirt/kubevirt/blob/abdfdb88efff132a65190c92dfffc630f4cab68f/cmd/virt-launcher/node-labeller/node-labeller.sh#L43 
virsh domcapabilities --machine $MACHINE --arch $ARCH --virttype $VIRTTYPE | virsh hypervisor-cpu-baseline --features /dev/stdin --machine $MACHINE --arch $ARCH --virttype $VIRTTYPE > /var/lib/kubevirt-node-labeller/supported_features.xml
👍 1
m
we have an enhancement under review to streamline the cpu model/feature discovery ux at https://github.com/harvester/harvester/pull/7937
it's something that has been affecting our users.. we try to work with upstream to streamline the ux but reception has been lukewarm
😭 1
b
There is no feature with ipred in that xml file on the node (or should I look into some container?). Not in Harvester 1.5.1, not in 1.6.1
c
it might be in one of the other xml files in that dir. idk, @millions-microphone-3535 and @great-bear-19718 are more hands on with this stuff than I am. I’m just reading code here 🤷
and like Ivan said this is coming from qemu -> libvirt -> kubevirt
g
is there a specific vm i can check which did not migrate.. i have the bundle
b
Nope. Did an case-insensitive grep. Maybe the name is different?
g
are you running grep on the cluster you upgraded or older one?
in the upgraded one its likely not present which is why the label is now missing
b
It's not present on both.
👍 1
All VMs in the SB are affected except the one with "-tlx6r" in the end.
👍 1
This one was stopped and restarted. Then the feature is not set in the VM pod nodeselector anymore and the VM can again be life migrated.
g
i need time to check
can you please also upload 
/var/lib/kubevirt-node-labeller/supported_features.xml
to the support case from one of the upgrade nodes and one of the older ones please?
b
Take your time. It was a test cluster. Nobody is crying yet, except me 🙂
c
… does the 1.5.1 cluster you’re looking at have that feature in the node labels?
g
i can see them in the support bundle
b
I added both files to the case.
👍 1
The other 1.5.1 cluster doesn't have that label. But it's a different CPU type. But I have more, I just need to find one with the exact same...
👍 1
g
thanks.. even i am trying to find if we have a node with exact cpu model as well
c
I’m just looking at newNodeLabeler()-> loadAll() -> loadHostSupportedFeatures() which reads from that XML to populate the usableFeatures list, which is then returned by getSupportedCPUFeatures(). That list is just ranged over, has the 
<http://cpu-feature.node.kubevirt.io/|cpu-feature.node.kubevirt.io/>
prefix added to it, and is then added as a node label.
👍 1
ok well if the other cluster doesn’t have that label then no surprise it’s not in the xml file…
g
yeah we need an older version of harvester with same cpu model
b
The first SB was created after the first node was upgraded. The other 5 nodes should have the old Harvester version. So maybe get all the SBs (there are plenty) to check the state before the upgrade. I'm continuing to look for an identical machine. Not that easy though...
👍 1
g
are you please also able to update the exact cpu model in the support case
i will also try and find them
b
Intel(R) Xeon(R) Gold 5418Y
👍 1
c
that would appear to be Sapphire Rapids?
b
Saphire Rapids-SP
Bad luck. No server with exactly the same CPU. I have more of the family, but no exactly the same.
Intel(R) Xeon(R) Gold 6426Y is the same familiy, but doesn't have the ipred label with Harvester 1.5.1
Sorry. No other machines with the same CPU family. There are just a few Dell R760 with that CPU.
👍 1
g
thanks a lot for your help
b
Thanks for your help!
g
i will also try and find this model in our lab
👍 1
b
I'm off to bed now. It's very late here. Checking back tomorrow.
c
I am a little confused, qemu didn’t even support that feature until 9.2.0 and harvester 1.5.1 has qemu 8.2.8: https://github.com/qemu/qemu/commit/10eaf9c0fb7060f45807becbb2742a9de9bc3632
b
Sorry, I can't comment that. I reported what I saw.
c
perhaps a case of SLE folks backporting patches
you might add 
cpuid
output to your case, not sure if that’s part of the bundle or not
b
For the versions: No big difference in kernel versions. Old: 5.14.21-150500.55.103-default New: 5.14.21-150500.55.121-default kubevirt and qemu are also different. 1.6.1 has kubevirt 1.5.2, 1.5.1 has kubevirt 1.4.0. qemu 8.2.8 in Harvester 1.5.1, 9.2.4 in Harvester 1.6.1
No 
cpuid
binary on the Harvester node.
dmidecode says:
Copy code
Processor Information
	Socket Designation: CPU2
	Type: Central Processor
	Family: Xeon
	Manufacturer: Intel
	ID: F8 06 08 00 FF FB EB BF
	Signature: Type 0, Family 6, Model 143, Stepping 8
	Flags:
		FPU (Floating-point unit on-chip)
		VME (Virtual mode extension)
		DE (Debugging extension)
		PSE (Page size extension)
		TSC (Time stamp counter)
		MSR (Model specific registers)
		PAE (Physical address extension)
		MCE (Machine check exception)
		CX8 (CMPXCHG8 instruction supported)
		APIC (On-chip APIC hardware supported)
		SEP (Fast system call)
		MTRR (Memory type range registers)
		PGE (Page global enable)
		MCA (Machine check architecture)
		CMOV (Conditional move instruction supported)
		PAT (Page attribute table)
		PSE-36 (36-bit page size extension)
		CLFSH (CLFLUSH instruction supported)
		DS (Debug store)
		ACPI (ACPI supported)
		MMX (MMX technology supported)
		FXSR (FXSAVE and FXSTOR instructions supported)
		SSE (Streaming SIMD extensions)
		SSE2 (Streaming SIMD extensions 2)
		SS (Self-snoop)
		HTT (Multi-threading)
		TM (Thermal monitor supported)
		PBE (Pending break enabled)
	Version: Intel(R) Xeon(R) Gold 5418Y
I don't know if you guys do backports for certain things. So version numbers can be "fuzzy".
g
we need to check a few things
right now we are trying to just collect as much info we think may be useful for our kernel team as well
👍 1
c
When were the affected VMs started? Before the upgrade from 1.5.1 to 1.6.1 was started?
b
Before the upgrade. All VM which were started after the upgrade (or stopped and started again) are life migratable.
57 Views
Open in Slack
PreviousNext
Powered by