https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rancher-desktop
a
This message was deleted.
w
so nested would refer to virtualization inside another virtual machine. the ISA would still remain the same when it comes to normal nested support. macOS does have the twist where rosetta can allow x86 ISA on aarch though i imagine it would be only in the first level VM. nested is super new to VZ so is it possible colima is using qemu and full virtualization?
c
I don’t believe colima is using qemu here, from the docs they only support nested virtualization when using 
vz
it is only supported on m3 macs and newer, so the docs link makes sense
The issue I’m running into is SVE 1/2 are being reported as a CPU feature in my Docker VM. It seems like this isn’t supported at the hardware level on my M4 resulting in a 
SIGILL
from the compiler I’m running in Docker. For whatever reason 
nestedVirtualization
is the only workaround I’ve found.
At least — I think this is whats going on. The only thing I know is for sure is nestedVirtualization works 😅
w
so RD uses lima in the back so did you try using override.yaml to send that param? if not in the 0/lima.yaml is the machine's config and you could add it there. guessing since sve may not be supported in a nested scenario it is masked off of the ISA.
c
Thanks for the tip, I’ll try this! Yeah — if in the nested container SVE isn’t available I assume this triggers a fallback. That would explain why everything works. I guess the real issue here is the container advertising this CPU feature but it not actually being supported on my M4. Using nested containers likely just circumvents this.
The overrides.yaml looks to be working but I am getting an error:
Copy code
{
  "level": "fatal",
  "msg": "nested virtualization is not supported on this device",
  "time": "2025-01-13T22:13:16+01:00"
}
It looks like it should be supported though: https://github.com/lima-vm/lima/pull/2530#issuecomment-2335110455
I’m guessing Colima is doing the same thing, could it be the version of Lima bundled with RD?
w
Yup guessing the same. Seeing something similar with qemu where the current dep doesn't work with the m4s
c
There is a docker image in the 0/config yaml, I’m guessing it’s not as easy as using a newer image tag there?
w
That would be the VM last I checked and not a docker tarball, but there are some other parts in play app may be a layer dump in there too
c
Got it, thanks for the help! Good to know, hopefully in a newer version of RD a Lima update will get this working
f
Rancher Desktop 1.17.0 is using Lima 1.0.2, which should have all the changes required for nested virtualization on M3+ CPUs (using macOS 15+). Can you share your 
override.yaml
file?
Actually, nm, the "nested virtualization is not supported on this device" error will only be generated when 
nestedVirtualization
in the Lima config is true.
👍 1
Ok, new theory: the lima version bundled with Rancher Desktop has been compiled on macOS 13, where colima will use the version from homebrew, which will have been compiled on macOS 15.
I'll try to build our version on macOS 15 as well, but will need to confirm that it will still work properly all the way back to macOS 12 before we can use it in 1.17.1...
c
Great insights! Thanks for the info Jan
My override file was just setting nestedVirtualization to true, it was otherwise empty
f
If you want to test this, you could install the build from my private fork at https://github.com/jandubois/rancher-desktop/actions/runs/12774579747/artifacts/2430306746 and see if that one allows nested virtualization on your machine. Unfortunately I don't have an M4 machine available for testing myself. That build will identify as 
1.10.0-4031-g....
because my repo doesn't have the recent tags. It means it is 4031 commits after the 1.10.0 release 🙂 It is based on the current 
main
of the official repo, but uses a custom build of Lima that has been compiled on macOS 15 instead of macOS 13.
You need to be logged into GitHub to be able to download assets from CI builds, and you need to unzip them after download. You will also need to clear the quarantine bit because the builds are not signed:
Copy code
xattr -rd com.apple.quarantine /Applications/Rancher\ <http://Desktop.app|Desktop.app>
If you do try this, then please let me know if this worked or not!
c
Thanks Jan, I’m happy to test but it doesn’t look like artifacts are public in your repo
f
Are you sure you are logged into GitHub?
c
ah 🤦 — I’m not in the right browser
f
c
got it now
👍 1
f
And then download 
Rancher Desktop.aarch64.dmg
from there
The Lima version seems to work fine on macOS 12 Intel, but I'm just starting to run a full set of integration tests on it, to make sure.
c
I’m running it now, 
1.10.0-4031-gb9c1ff4c7
👍 1
f
The only question I have is if 
nestedVirtualization
works with that build or not
Via the 
override.yaml
mechanism
c
I think it does, at least I’m not seeing anything in lima stderr
let me see if I can dig up some logs to confirm it
"nestedVirtualization":true
I found this in lima.log
f
The important part would be not finding 
nested virtualization is not supported on this device
anywhere 🙂
c
with the current build of RD lima was failing to start
I’m going to see if I can reproduce the original SIGILL
f
Maybe I'm mixing this up; I thought it was only failing to run a container with the Java compiler?
c
Yeah without nestedVirtualization but enabling it via the overrides yml was causing RD to fail to start
f
Ah, ok, got it
c
let me check the container though, if this works I’m fairly confident nestedVirtualization is working
f
Did you switch to VZ mode, or are you running with QEMU?
c
Yeah I should be in vz mode
👍 1
f
I have one other idea you could try if you want to use the release version (but it will only work with QEMU):
Copy code
❯ cat override.yaml
cpuType:
  aarch64: cortex-a72
That should override the CPU that QEMU emulates from 
host
to a generic arm64 CPU. That might also avoid the Java compiler issue
c
I’m rebuilding my image right now to try this out
VZ would be nicer because QEMU is pretty slow for me
f
This is what Lima 1.0.3 does internally do deal with the SME issue on M4 CPUs. Rancher Desktop doesn't need to do that because it ships with a patched version of QEMU that disables the SME bit (the patch from UTM)
Yeah, of course. We should make VZ the default setting soon.
c
Yeah, it seems like nested virtualization kind of just fixes this issue by accident from what I understand
f
But just presenting options. I'll see if we will update to the Lima built on macOS 15 for the 1.17.1 release. But normally we only want to address regressions, and since nested virtualization is not a feature exposed by Rancher Desktop this is not really a regression.
But we'll see. First it must clear integration tests on both Intel and Arm...
c
yeah makes sense, colima is always an option for now
f
Yeah, colima just uses whatever is the latest Lima from homebrew
And homebrew builds bottles for each macOS version, so it is always compiled on the OS it runs on.
c
yeah
f
I'm somewhat surprised that Lima compiled on macOS 15 seems to run fine on macOS 12
I mean, we do set the min required version for Xcode etc, but still. Not something you could make work on Linux
c
Hm — this is from my container
Copy code
bash-5.2# /usr/lib/graalvm/bin/java --version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000ffff9a7d8ce8, pid=15, tid=16
let me double check the config
cpuType
has no impact with 
vz
correct?
f
(this Slack is misconfigured; I expected it to show the quoted code here)
c
I’m using a different mount type in my colima config, that’s about it
f
I don't know; mount type doesn't sounds like it should make a difference
c
it doesn’t but I think it did…
Copy code
bash-5.2# /usr/lib/graalvm/bin/java --version
openjdk 23.0.1 2024-10-15
OpenJDK Runtime Environment GraalVM CE 23.0.1+11.1 (build 23.0.1+11-jvmci-b01)
OpenJDK 64-Bit Server VM GraalVM CE 23.0.1+11.1 (build 23.0.1+11-jvmci-b01, mixed mode, sharing)
That really doesn’t make sense
f
You changed the mount type, and now it works?
c
Yeah
let me change it back
Maybe it was just restarting RD that dod it
f
was going to suggest exactly that 🙂
Which mount type worked, and which one breaks (if this is reproducible)?
c
They both work
I’m not sure what happened there
I’m guessing I saved my overrides file somehow after it was loaded
f
Me neither, but the base assumption has been confirmed: you need to compile Lima on macOS 15 to get support for nested virtualization
👍 1
c
Yup
f
Which makes sense, given that earlier SDKs may not have the right supporting stuff for it
Remember that the 
override.yaml
file will be deleted by a factory-reset!
c
Ok — it “works” as expected on my end now With 
nestedVirtualization: false
I’m back to getting the SIGILL
Copy code
bash-5.2# /usr/lib/graalvm/bin/java --version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000ffffa27d8ce8, pid=15, tid=16
f
Thanks for the confirmation!
I'll see if this can make it into 1.17.1, but no promises. Will be in 1.18.0 for sure, but I hope by then either Apple or Oracle have created a fix themselves.
c
Yeah, I’ve opened a but with Graal but I’m not even sure if this is an issue with them — it feels like Apple will need to fix this
f
I think so too, but who knows...
c
Though Graal will let you disable SVE but not SVE2 leading to a very annoying situation where things are half broken
f
Personally I subscribe to the sentiment from Larry Wall (Perl): "The Golden Gate wasn't our fault either, but we still built a bridge across it."
🤣 1
c
Thanks for your help here!
48 Views
Open in Slack
PreviousNext
Powered by