https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rancher-desktop
a
This message was deleted.
w
@bored-farmer-36655
b
@plain-fireman-25122 Hi, it's failing at the moment, think the switch from alpine to ubuntu. Haven't investigated further.
@plain-fireman-25122 is this the error you see?
Copy code
'time="2024-09-06T09:13:01-05:00" level=info msg="[hostagent] Waiting for the essential requirement 1 of 4: \\"ssh\\""\n' +
    'time="2024-09-06T09:13:01-05:00" level=debug msg="[hostagent] executing script \\"ssh\\""\n' +
    'time="2024-09-06T09:13:01-05:00" level=debug msg="[hostagent] executing ssh for script \\"ssh\\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\\"/home/username/.local/share/rancher-desktop/lima/_config/user\\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\\" -o User=username -o ControlMaster=auto -o ControlPath=\\"/home/username/.local/share/rancher-desktop/lima/0/ssh.sock\\" -o ControlPersist=yes -p 46733 127.0.0.1 -- /bin/bash]"\n' +
    'time="2024-09-06T09:14:16-05:00" level=debug msg="[hostage'... 5485 more characters,
  code: 1,
  [Symbol(child-process.command)]: '/opt/rancher-desktop/resources/resources/linux/lima/bin/limactl --debug start --tty=false 0'
}
2024-09-06T14:20:00.941Z: Progress: errored Starting Backend: Error: /opt/rancher-desktop/resources/resources/linux/lima/bin/limactl exited with code 1
@plain-fireman-25122 seems to me to be some sort of ssh issue, no ssh.sock is created? @fast-garage-66093 Hi, how to debug this issue further?
p
@bored-farmer-36655 That looks like it. Fails to get a ready response from lima backend and the errors all look like ssh connection issues. I'm starting it back up to get the exact error messages again.
Yup, that's the error.
f
The 
5485 more characters
part of the log is unexpected. Is it truncated like this in the log file itself?
p
D'oh, just did a factory reset. Give me a couple minutes. Which log file you looking for?
b
@fast-garage-66093 Hi, not sure, which log would this be in? I started now from command line with verbose option and see this;
Copy code
{"level":"debug","msg":"stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\nConnection reset by 127.0.0.1 port 46545\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\nConnection reset by 127.0.0.1 port 46545\\r\\n\": exit status 255","time":"2024-09-06T10:53:59-05:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2024-09-06T10:54:09-05:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2024-09-06T10:54:09-05:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/home/username/.local/share/rancher-desktop/lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=username -o ControlMaster=auto -o ControlPath=\"/home/username/.local/share/rancher-desktop/lima/0/ssh.sock\" -o ControlPersist=yes -p 46545 127.0.0.1 -- /bin/bash]","time":"2024-09-06T10:54:09-05:00"}
f
This is different from the previous log you posted. What do you mean by "started from command line"?
b
rdctl start --verbose, but it does pop up the same error in the GUI dialog box
f
Ah, ok, just wanted to make sure you are not invoking 
limactl
manually, which can add all manner of issues
b
Ahh it's in the lima.log
f
I kind of doubt it, but is there any additional error message in the 
ha.stderr.log
?
b
Well just the startup and then multiple entries of the above debug message until it pops up the GUI dialog error
Copy code
cat lima.ha.stderr.log
 
{"level":"debug","msg":"Creating iso file /home/username/.local/share/rancher-desktop/lima/0/cidata.iso","time":"2024-09-06T10:46:53-05:00"}
{"level":"debug","msg":"Using /tmp/diskfs_iso1483483279 as workspace","time":"2024-09-06T10:46:53-05:00"}
{"level":"debug","msg":"OpenSSH version 9.8.1 detected","time":"2024-09-06T10:46:53-05:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing <mailto:aes128-gcm@openssh.com|aes128-gcm@openssh.com> and <mailto:aes256-gcm@openssh.com|aes256-gcm@openssh.com>","time":"2024-09-06T10:46:53-05:00"}
{"level":"info","msg":"hostagent socket created at /home/username/.local/share/rancher-desktop/lima/0/ha.sock","time":"2024-09-06T10:46:53-05:00"}
{"level":"debug","msg":"Start udp DNS listening on: 127.0.0.1:50600","time":"2024-09-06T10:46:53-05:00"}
{"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:33197","time":"2024-09-06T10:46:53-05:00"}
{"level":"debug","msg":"QEMU version 9.0.2 detected","time":"2024-09-06T10:46:54-05:00"}
{"level":"debug","msg":"firmware candidates = [/home/username/.local/share/qemu/edk2-x86_64-code.fd /usr/share/qemu/edk2-x86_64-code.fd /usr/share/OVMF/OVMF_CODE.fd /usr/share/OVMF/OVMF_CODE_4M.fd /usr/share/qemu/ovmf-x86_64-code.bin /usr/share/edk2-ovmf/x64/OVMF_CODE.fd]","time":"2024-09-06T10:46:54-05:00"}
{"level":"info","msg":"Using system firmware (\"/usr/share/qemu/ovmf-x86_64-code.bin\")","time":"2024-09-06T10:46:54-05:00"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/home/username/.local/share/rancher-desktop/lima/0/serial*.log\")","time":"2024-09-06T10:46:54-05:00"}
{"level":"debug","msg":"qCmd.Args: [/usr/bin/qemu-system-x86_64 -m 6144 -cpu host -machine q35,accel=kvm -smp 2,sockets=1,cores=2,threads=1 -drive if=pflash,format=raw,readonly=on,file=/usr/share/qemu/ovmf-x86_64-code.bin -boot order=d,splash-time=0,menu=on -drive file=/home/username/.local/share/rancher-desktop/lima/0/basedisk,format=raw,media=cdrom,readonly=on -drive file=/home/username/.local/share/rancher-desktop/lima/0/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/home/username/.local/share/rancher-desktop/lima/0/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:46545-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:66:5b:95 -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/home/username/.local/share/rancher-desktop/lima/0/serial.sock,server=on,wait=off,logfile=/home/username/.local/share/rancher-desktop/lima/0/serial.log -serial chardev:char-serial -chardev socket,id=char-serial-virtio,path=/home/username/.local/share/rancher-desktop/lima/0/serialv.sock,server=on,wait=off,logfile=/home/username/.local/share/rancher-desktop/lima/0/serialv.log -device virtio-serial-pci,id=virtio-serial0,max_ports=1 -device virtconsole,chardev=char-serial-virtio,id=console0 -chardev socket,id=char-qmp,path=/home/username/.local/share/rancher-desktop/lima/0/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -chardev socket,path=/home/username/.local/share/rancher-desktop/lima/0/ga.sock,server=on,wait=off,id=qga0 -device virtio-serial -device virtserialport,chardev=qga0,name=io.lima-vm.guest_agent.0 -name lima-0 -pidfile /home/username/.local/share/rancher-desktop/lima/0/qemu.pid]","time":"2024-09-06T10:46:54-05:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2024-09-06T10:46:54-05:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2024-09-06T10:46:54-05:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/home/username/.local/share/rancher-desktop/lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=username -o ControlMaster=auto -o ControlPath=\"/home/username/.local/share/rancher-desktop/lima/0/ssh.sock\" -o ControlPersist=yes -p 46545 127.0.0.1 -- /bin/bash]","time":"2024-09-06T10:46:54-05:00"}
p
same when running 
rancher-desktop
f
Sorry, I have no idea. This is with 1.15.1 on openSUSE?
p
Yep.
Version : 1.15.1-lp153.3.3
f
@rapid-eye-50641 Which Linux version are you using for testing?
p
NAME="openSUSE Tumbleweed" # VERSION="20240904" ID="opensuse-tumbleweed" ID_LIKE="opensuse suse" VERSION_ID="20240904"
f
@plain-fireman-25122 @bored-farmer-36655 How did you install Rancher Desktop, as an AppImage, or from the repo?
p
from the repo
Copy code
more /etc/zypp/repos.d/isv_Rancher_stable.repo 
[isv_Rancher_stable]
name=Rancher stable packages (rpm)
enabled=1
autorefresh=0
baseurl=<https://download.opensuse.org/repositories/isv:/Rancher:/stable/rpm/>
gpgcheck=1
gpgkey=<https://download.opensuse.org/repositories/isv:/Rancher:/stable/rpm/repod>
ata/repomd.xml.key
r
I test on Ubuntu 22.04.4
b
likewise repository version and on the same Tumbleweed snapshot 20240904.
p
1.15.0-lp153.2.3 worked as did 1.15.1-lp153.3.1
Just downloaded the app image and that is starting.
While the appimage works, the repo and rpm packages are far more convenient with multiple systems/users with management system in place.
f
We are going to try to test/repro this ourselves
p
Fresh install of Tumbleweed, refresh dup --no-allow-vendor-change; reboot; add rancher repo and install, reboot for good measure.
b
@plain-fireman-25122 not an option here on my development machine 😉 So that did sort it out for you?
p
No, that's to replicate.
The app image does work which is getting me by for right now but repo solution would be optimal.
I have two Tumbleweed systems. One is my main dev system, the other is a laptop I did a fresh install on just to make sure I didn't do something silly on my main dev box.
b
@plain-fireman-25122 ahh ok 😄 I updated to 20240905, had to 
rm -rf ~/.local/share/rancher-desktop/lima/0
to clear the image to get it to start, but still same error.
p
yep. I've done manual cleanup and also clicked on the factory reset from the troubleshooting window.
I've also reverted the package version in the repo but unfortunately, the last package that worked isn't available in the repo. Tried 1.14 package in the repo, which also fails.
f
The only explanation I have is that some dependency must have been updated, so OBS rebuilt the package, which now no longer works.
p
That's what I'm thinking. I know the 1.15.0 package worked but it's not available anymore that I can tell.
f
There is just a single 
1.15
package definition in OBS, so any rebuild will drop the previous version. So you only have the latest build of each minor version available, and not of each patch release
p
Not sure if this helps any, but I'm pretty sure 1.15.1-lp153.3.1 worked. This was around time I took off for vacation so not 100%:
Copy code
2024-08-10 13:10:29|install|rancher-desktop|1.15.0-lp153.2.3|x86_64||isv_Rancher_stable|197d4ccab734f7ca6de9577ccab396baf7686b241b920133c7ce3ed9c0d072ee|
2024-08-16 16:46:41|install|rancher-desktop|1.15.1-lp153.3.1|x86_64||isv_Rancher_stable|03bc0faa12184cc25c7d9b5da62515158a4444c823f466fc043f62a7ebccc595|
2024-08-24 12:35:01|install|rancher-desktop|1.15.1-lp153.3.2|x86_64||isv_Rancher_stable|0c276b0496e4107796c446a9015216211d8fa50f0df4fe029239779136ffade9|
2024-08-26 13:54:59|command|root@river|'zypper' 'rm' '-u' 'rancher-desktop'|
2024-08-26 13:55:00|remove |rancher-desktop|1.15.1-lp153.3.2|x86_64|root@river|
2024-08-26 13:59:00|command|root@river|'zypper' 'in' 'rancher-desktop'|
1.15.1-lp153.3.2 is when I definitely noticed I was having issues.
1.14.2-lp153.5.7 worked, however 1.14.2-lp153.5.11 does not.
f
@proud-jewelry-46860 found that by downgrading to https://download.opensuse.org/history/20240813/tumbleweed/repo/oss/noarch/qemu-ovmf-x86_64-202402-1.1.noarch.rpm you can get Rancher Desktop 1.15.1 working again on Tumbleweed.
Copy code
sudo zypper in --oldpackage qemu-ovmf-x86_64-202402-1.1.noarch.rpm
p
Interesting - I have have kvm images that are working as expected.
p
Also 
…-202402-2.1
doesn't work either. At this point somebody needs to figure out if that can be reproduced with a more generic image, etc.
I don't see anything in bugzilla that looks similar that isn't older than the date ranges involved.
b
@fast-garage-66093 thanks for the I guess temporary fix, it's all working now. Seems those new changes relate to secure boot?
p
Yeah, given that 
-2.1
also looks broken, https://build.opensuse.org/request/show/1193325 is probably involved in some way
f
@bored-farmer-36655 just hang on to that older RPM in case it is wiped from the history archives too
I wonder what this means:
```Removed features tag:
"acpi-s3", "requires-smm", "secure-boot", "enrolled-keys"```
b
@fast-garage-66093 yes, 
zypper al qemu-ovmf-x86_64
as well 😉
p
@fast-garage-66093 are you going to file a bug? (Probably involves reducing it to not need lima to repro…)
f
idk yet, I'm trying... 🙂
I can repro with just Lima, but we probably need something even simpler
p
Can confirm qemu-ovmf-x86_64|202402-1.1 is working on my system
f
I've filed https://bugzilla.opensuse.org/show_bug.cgi?id=1230291 in case you want to track this
b
@fast-garage-66093 thanks for this, yup added myself 😉
i
Copy code
<https://download.opensuse.org/history/20240813/tumbleweed/repo/oss/noarch/qemu-ovmf-x86_64-202402-1.1.noarch.rpm>
says: Resource is no longer available!
b
f
@bored-farmer-36655 Looks like the latest Tumbleweed updates made it work again. I've unpinned 
qemu-ovmf-x86_64
and upgraded everything, and Rancher Desktop still seems to work fine.
We have updated Lima to pick the combined firmware file in the future, but even for the Rancher Desktop 1.16.0 release everything seems to be working again. Let me know if this is not the case for you.
p
That as of today? I tried Friday and still broken..
Will dup tonight and let you know.
f
I tried it just now, but I don't see any update since Oct 9 on https://build.opensuse.org/projects/openSUSE:Factory/packages/ovmf/files/ovmf.changes?expand=1, so it should have been the same on Friday...
I've bumped Lima in Rancher Desktop to 1.0.0-beta.0 to test again, but decided to just test Rancher Desktop 1.16.0 again before, and it also just worked. 🤷
p
That's what I thought. I'll dup again and see tonight. 202405-4.1 current version?
Have a few qemu updates for today. Let you know after a dup.
f
Copy code
Version        : 202405-4.1
p
Yeah, that's what's on my system. Dup'ng right now and reboot once it's done and let you know.
f
Thanks! Maybe the fix is in QEMU itself, that it fakes up an empty VARS mount if you only give it the CODE file?
p
we'll see 😉
f
I thought it was weird that it worked like this on all other distros, but could not be made to work for Tumbleweed...
p
Yeah, been following that one and lima issue 2647 and been waiting to see when that got promoted.
b
@fast-garage-66093 just checking 😉
p
rebooting...
f
Lima issue 2647 is merged and will be in future Rancher Desktop releases, but doesn't help 1.16.0.
But right now it looks to me like the Lima change wouldn't be needed (but we'll leave it in place, as long as it continues to work)
p
not looking good. Taking too long to start up the virtual machine...
f
Weird, I wonder why it works for me
I'm running in a VM, and just did a factory reset. When starting again I get "Starting backend" after 45s, and everything is running and the progress bar gone around 55s in.
p
That's what I would expect. I haven't done a factory reset. Will try that next.
f
Copy code
jan@localhost:~> zypper lu
Loading repository data...
Reading installed packages...
No updates found.
jan@localhost:~> zypper ll

There are no package locks defined.
p
Copy code
All repositories have been refreshed.
river(5):> zypper lu
Loading repository data...
Reading installed packages...
No updates found.
river(6):> zypper ll

There are no package locks defined.
Running from cmd-line:
Copy code
river(25):> rancher-desktop
[5386:1021/153137.331052:ERROR:<http://connection.cc|connection.cc>(711)] Cannot send request of length 18215856
f
Rebooting now
p
Just error'd with same error for ssh.
I'll try factory reset.
b
Yup failed here too 😢 will reset and reboot too...
f
Unchanged after reboot; everything works
And this is 1.16.0 downloaded from the GitHub release page
p
Ahhh, I've got 1.16.0 from the repo.
Copy code
river(16):> zypper info rancher-desktop
Loading repository data...
Reading installed packages...


Information for package rancher-desktop:
----------------------------------------
Repository     : Rancher stable packages (rpm)
Name           : rancher-desktop
Version        : 1.16.0-lp153.2.4
Arch           : x86_64
Vendor         : <obs://build.opensuse.org/isv:Rancher>
Installed Size : 1.24 GiB
Installed      : Yes
Status         : up-to-date
Source package : rancher-desktop-1.16.0-lp153.2.4.src
Upstream URL   : <https://github.com/rancher-sandbox/rancher-desktop#readme>
Summary        : Kubernetes and container management on the desktop
Description    : 
    Rancher Desktop is an open-source project to bring Kubernetes and container
    management to the desktop
f
That should have been the same code; I guess I should try that
p
Not looking good for the factory reset either 😞
f
I noticed that I had a newer Lima version installed, and removed it. It shouldn't make a difference because Rancher Desktop should only use the bundled version. And it didn't make a difference. So I guess trying the repo version is the next step for me too
Wild, the repo version doesn't work for me either. And I now see the warning/error
Copy code
[7192:1021/145039.658883:ERROR:<http://connection.cc|connection.cc>(711)] Cannot send request of length 18215856
I didn't see this with the version from GitHub.
p
The appimage version works as well. Was going to run that next to see if the connection.cc error pops up with that one but waiting for the repo version to fail out.
f
The appimage version always worked because it bundles its own version of QEMU
p
And has the same connection.cc error.
just started it up.
f
But the GitHub zip file and the RPM from the repo should have been functionally identical. Of course the RPM is rebuilt from source inside OBS, so is not really the same, but otherwise should behave the same
Anyways, thanks for letting me know. It means I can't really just test new releases by installing them from GitHub.
But I'm curious to understand why the GitHub release works.
p
I'll pull down the zip file and see if it works here.
f
So I guess I get to spend the afternoon on that instead of just being able to close the issue as "fixed" 😞
p
Sorry - know how that feels
f
I wish we wouldn't have to deal with OBS at all, but I don't see how we can release rpm/deb files without it.
p
Not sure if this is of any comfort but zip file from github is working fine for me as well
b
@fast-garage-66093 yes, I'm using the rpm as well... If you trigger the service on OBS, it should pull the zip file down?
f
No, I downloaded 
rancher-desktop-linux-v1.16.0.zip
from https://github.com/rancher-sandbox/rancher-desktop/releases/tag/v1.16.0
And then I ran 
rm -rf /opt/rancher-desktop/*
and unzipped it in there
Just tried it again, and it works (after a factory-reset)
b
@fast-garage-66093 Build log looks fine on OBS, are you removing the lima resources?
Copy code
# Remove qemu binaries included in lima tarball
rm -v resources/resources/linux/lima/bin/qemu-* 
rm -rvf resources/resources/linux/lima/lib
rm -rvf resources/resources/linux/lima/share/qemu
f
No, I don't. That explains it; the GitHub ZIP file includes the QEMU files for the AppImage build.
Thanks, that give me a way to test the latest GitHub builds without going through OBS
And I can confirm that removing the bundled QEMU resources will break the Rancher Desktop version from GitHub too
I guess this points out another workaround though: if you don't want to lock the 
qemu-ovmf-x86_64
package from 
zypper
, then you could uninstall the 
rancher-desktop
package and install it from GitHub with the bundled QEMU that would only be used by the Lima version bundled with Rancher Desktop. Not great either, but an alternative.
Especially if you don't have the old 
qemu-ovmf-x86_64
package anymore. 😄
b
True 😄
f
Ok, the latest CI build with Lima-1.0.0-beta.0 seems to still run even after deleting all QEMU resources from 
/opt/rancher-desktop
, so I guess this is still good.
Thanks for all your help!
b
your very welcome 😄
@fast-garage-66093 FYI, just tried the updated qemu-ovmf-x86_64 202408-1.1 no joy so rolled back to the 202402-1.1 version.
f
Thank you for testing. It failed for me too when I deleted the QEMU binaries that are included in the GitHub ZIP file. Using that file is more like using the AppImage (and the bundled QEMU from the ZIP is where the AppImages gets its binaries from).
76 Views
Open in Slack
PreviousNext
Powered by