https://rancher.com/ logo
Powered by
Title
q

quick-keyboard-83126

03/22/2023, 6:13 PM
So, we're hunting down a problem where 
docker pull <http://gcr.io/|gcr.io/>...
or 
nerdctl pull <http://gcr.io/|gcr.io/>...
is slow when their daemon is Rancher Desktop on macOS. We've been able to isolate it to a 
curl
w/ credentials from 
gsutil
inside 
rdctl shell
... I'm wondering if someone would be up for trying a command out for me...
f

fast-garage-66093

03/22/2023, 10:06 PM
I'm happy to try something... 🙂
q

quick-keyboard-83126

03/22/2023, 10:07 PM
Gimme a minute, I'm trying to finish a 
bun
change
f

fast-garage-66093

03/22/2023, 10:07 PM
I have to review something here right now, so won't be able to try for 20-30min anyways
q

quick-keyboard-83126

03/22/2023, 10:11 PM
So, the command is basically something like this, to be run in 
rdctl shell
where 
~/gcloud/google-cloud-sdk
is an installation of 
gcloud
and you've done 
apk add py3-openssl
or similar, and you've created a 
json
file at 
$service_account_key
which is a google cloud service account (w/ enough permissions to identify and make cloud storage read operations I suppose....) 
curl "$(~/gcloud/google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abf>
f2a6d853109e4a948| tail -n 1 |awk '{print $5}')" -D - -o /dev/null
I know that's a lot of prerequisites, sorry...
Err, be very careful about copy paste, each time I try I end up w/ a stupid line wrap in the url which explodes when copied into the shell (!!) 
curl "$(~/gcloud/google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948%7C> tail -n 1 |awk '{print $5}')" -D - -o /dev/null
Sigh, I hate copy/paste corruption... here's a version where my pipe isn't inexplicably percent encoded: 
curl "$(~/gcloud/google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092>
cb075baa95784488c0bbc3e0abff2a6d853109e4a948 | tail -n 1 |awk '{print $5}')" -D - -o /dev/null
And here's what it looks like when I run it 
lima-rancher-desktop:/Users/jsoref$ curl --http1.1 -4 "$(~/gcloud/google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948> | tail -n 1 |awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
X-GUploader-UploadID: ADPycdvrNdaWfDDMITCajmstn98LaD4n43bMnn3VPmf4-AIVxmbmr_y-h7MY_KNTngpIRmv_qkJLyk8N6Sh0gNyvwyNAvg
Expires: Wed, 22 Mar 2023 23:14:37 GMT
Date: Wed, 22 Mar 2023 22:14:37 GMT
Cache-Control: public, max-age=3600
Last-Modified: Mon, 06 Mar 2023 23:41:48 GMT
ETag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
Content-Type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 9599350
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

 16 9374k   16 1552k    0     0  23989      0  0:06:40  0:01:06  0:05:34  5607
Note that it more or less never finishes... The same equivalent command is fine when not run in the rancher lima vm
f

fast-garage-66093

03/22/2023, 10:24 PM
Unfortunately I don't have a GCloud account ready for testing, so I can't actually run it.
q

quick-keyboard-83126

03/22/2023, 10:25 PM
20 9374k 20 1930k 0 0 3099 0 0:51:37 0🔟37 0:41:00 272
f

fast-garage-66093

03/22/2023, 10:25 PM
But in the end this is just a 
curl
command, so if this is a client-side problem, it should be reproducible with a curl to some other endpoint
q

quick-keyboard-83126

03/22/2023, 10:25 PM
fwiw, it's still running (and will basically not finish, it'll die when i leave the office)
we don't see this problem w/ curl from outside rancher and we don't see this problem pulling from other container repositories (mostly docker hub, but also others)
f

fast-garage-66093

03/22/2023, 10:26 PM
So I would try to find out if this is a specific networking bottleneck, and if it depends on the endpoint on the otherside or not
q

quick-keyboard-83126

03/22/2023, 10:26 PM
Right, but,... it works fine from outside rancher w/ the same exact command
I'm also talking w/ google (via our vendor)...
It doesn't matter if i add 
-4
or 
--http1.1
, i'm not sure what else i can do to get interesting bits out
i could set up some packet traces and compare i suppose
(i'm not a fan of trying to decipher https traffic...)
f

fast-garage-66093

03/22/2023, 10:29 PM
Are you also running plain Lima? If you are, then I would try to start both an Alpine and a Ubuntu VM, and try the same thing from both VMs, to determine if this is an issue with the distro
q

quick-keyboard-83126

03/22/2023, 10:29 PM
i can run anything someone tells me to run w/in reason
hmm, are you suggesting using 
docker run
to run an alpine/ubuntu in the rancher managed thing? or to find the standalone lima and try that?
f

fast-garage-66093

03/22/2023, 10:30 PM
If you don't have Lima, use 
brew install lima
. Then 
limactl start --tty=false
to create a Ubuntu VM, and 
limactl start --tty=false <template://alpine>
to create an Alpine one
q

quick-keyboard-83126

03/22/2023, 10:31 PM
ok
Error: lima HEAD-cce3561 is already installed
To install 0.15.0, first run:
  brew unlink lima
f

fast-garage-66093

03/22/2023, 10:31 PM
Oh, so you already have it, but switching to the latest release is probably a good idea
q

quick-keyboard-83126

03/22/2023, 10:32 PM
so unlink and install again?
f

fast-garage-66093

03/22/2023, 10:32 PM
Yes
q

quick-keyboard-83126

03/22/2023, 10:32 PM
🍺
f

fast-garage-66093

03/22/2023, 10:32 PM
🍻
Now you can create the 2 VMs
Then use 
lima
to run a shell in the Ubuntu VM, and 
limactl shell alpine
to run one in the Alpine one
I don't know if you need to write to your 
$HOME
from inside the VM. If you do, then you need to edit the 
lima.yaml
before start (don't use 
--tty=false
) because the templates will mount it readonly
q

quick-keyboard-83126

03/22/2023, 10:35 PM
i only need a tiny json that i can paste into the vm
==> Pouring qt--6.4.3.arm64_ventura.bottle.tar.gz
f

fast-garage-66093

03/22/2023, 10:35 PM
Both VMs will have 
/tmp/lima
mounted writable
q

quick-keyboard-83126

03/22/2023, 10:36 PM
mounted from the host?
f

fast-garage-66093

03/22/2023, 10:36 PM
Yes
q

quick-keyboard-83126

03/22/2023, 10:36 PM
% lima --version
limactl version 0.15.0
👍 1
f

fast-garage-66093

03/22/2023, 10:37 PM
I have no idea why it installed 
qt
as well...
q

quick-keyboard-83126

03/22/2023, 10:37 PM
INFO[0019] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0025] READY. Run 
limactl shell alpine
to open the shell.
I really wish that there was an easy way to install 
gcloud
w/ 
apk
f

fast-garage-66093

03/22/2023, 10:38 PM
Good. I would expect you to get similar performance from the Alpine VM as you get from Rancher Desktop; this is just to make sure the issue doesn't come from something else RD is doing.
And then I'm really curious if Ubuntu shows the same behaviour or not
q

quick-keyboard-83126

03/22/2023, 10:40 PM
ubuntu is still setting up
f

fast-garage-66093

03/22/2023, 10:41 PM
I'll let you repeat your experiments here and get some coffee. Should be back in 10-15min
q

quick-keyboard-83126

03/22/2023, 10:43 PM
not finishing instantly: 
lima-alpine:/tmp/google-cloud-sdk$ curl "$(./bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948%7C> ta
il -n 1 |awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/2 200
x-guploader-uploadid: ADPycdvTI4bzssRZPSewhhm1teYh9Cb7NziZPUyVDKe2j6ui2vIBI0JbDrKC6FxUlSYehdbsl22xsXQmoxpzoNl11jumVw
expires: Wed, 22 Mar 2023 23:42:49 GMT
date: Wed, 22 Mar 2023 22:42:49 GMT
cache-control: public, max-age=3600
last-modified: Mon, 06 Mar 2023 23:41:48 GMT
etag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
content-type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 9599350
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

  6 9374k    6  575k    0     0  32581      0  0:04:54  0:00:18  0:04:36 24920
INFO[0419] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
ubuntu is still twiddling its thumbs
INFO[0579] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
FATA[0600] did not receive an event with the "running" status
jsoref@lima-default:/Users/jsoref$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 21.10
Release:	21.10
Codename:	impish
um, i was able to copy stuff to/from /tmp/lima from the alpine shell, but the ubuntu shell doesn't see any files in /tmp/lima ...
f

fast-garage-66093

03/22/2023, 10:53 PM
Weird. Did the Ubuntu install eventually finish?
q

quick-keyboard-83126

03/22/2023, 10:53 PM
dunno, it was eventually running
f

fast-garage-66093

03/22/2023, 10:54 PM
I think it means 
sshfs
was not yet installed. Do you see the files from your home directory? I would expect that to be missing as well then
q

quick-keyboard-83126

03/22/2023, 10:55 PM
also missing
thankfully none of that is necessary for this adventure
f

fast-garage-66093

03/22/2023, 10:55 PM
Maybe try 
limactl stop; limactl start
to see if it fixes itself?
q

quick-keyboard-83126

03/22/2023, 10:55 PM
jsoref@lima-default:/tmp$ ./google-cloud-sdk/bin/gsutil signurl x <gs://foo>
Keystore password:
well gee, that's helpful ...
I have no idea who is asking for a keystore password or why, but i know I don't like it 😞
f

fast-garage-66093

03/22/2023, 10:56 PM
Must be 
gsutil
, no?
q

quick-keyboard-83126

03/22/2023, 10:56 PM
well, sure, but it didn't do that on macOS, nor in rdctl nor in lima-alpine
why now?
f

fast-garage-66093

03/22/2023, 10:56 PM
I have no idea
LOL macos - Keystore password? Huh? - Stack Overflow
q

quick-keyboard-83126

03/22/2023, 10:58 PM
I'm w/ the original reporter of that item
note: the file 
x
was an empty file
it would have been better for it to say "there doesn't appear to be anything signed here,..."
jsoref@lima-default:/tmp$ curl "$(google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948%7C> tail -n 1 | awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/2 200
x-guploader-uploadid: ADPycdvXfd5Ls6eGM-shkeomb6v7Jl1wM-SLroVEfTj5fe2UoDjG3n-FvhDq8nXeoNZA-L4ECU6yxpvsocpdF_RoxlWe0A
expires: Wed, 22 Mar 2023 23:59:39 GMT
date: Wed, 22 Mar 2023 22:59:39 GMT
cache-control: public, max-age=3600
last-modified: Mon, 06 Mar 2023 23:41:48 GMT
etag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
content-type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 9599350
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

  7 9374k    7  668k    0     0  48877      0  0:03:16  0:00:14  0:03:02 65298
ok, lima-default(=ubuntu) is behaving like lima-alpine
f

fast-garage-66093

03/22/2023, 11:00 PM
which is behaving like lima-rancher-desktop, right?
q

quick-keyboard-83126

03/22/2023, 11:01 PM
right
I can try 
UTM
...
f

fast-garage-66093

03/22/2023, 11:01 PM
something something MTU
q

quick-keyboard-83126

03/22/2023, 11:01 PM
hmm, I'd buy MTU
lemme see what UTM has to say
f

fast-garage-66093

03/22/2023, 11:02 PM
I remember @wide-mechanic-33041 talked about having to adjust the MTU settings in some cases to get the network to work properly, but I can't remember any details
q

quick-keyboard-83126

03/22/2023, 11:03 PM
UTM says "here's an ubuntu, I hope you remember its credentials". Me. "Um, that's a negative"
f

fast-garage-66093

03/22/2023, 11:08 PM
Did you try to change MTU to see if that makes a difference?
q

quick-keyboard-83126

03/22/2023, 11:08 PM
Don't know how to 🙂
f

fast-garage-66093

03/22/2023, 11:08 PM
E.g. try something quite low, like 
ifconfig eth0 mtu 1000
and see if that changes anything?
q

quick-keyboard-83126

03/22/2023, 11:08 PM
jsoref@lima-default:/tmp$ sudo ifconfig eth0 mtu 1000
sudo: ifconfig: command not found
🙂
lemme google...
f

fast-garage-66093

03/22/2023, 11:09 PM
Try it with RD or Alpine first
we kind of have established that the issue is not the distro, so we can stop testing on Ubuntu, I think
q

quick-keyboard-83126

03/22/2023, 11:10 PM
lima-alpine:/Users/jsoref$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:af:99:94 brd ff:ff:ff:ff:ff:ff
lima-alpine:/Users/jsoref$ sudo ip link set dev eth0 mtu 1000
lima-alpine:/Users/jsoref$ ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1000 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:55:55:af:99:94 brd ff:ff:ff:ff:ff:ff
f

fast-garage-66093

03/22/2023, 11:10 PM
Ok, now do the 
curl
again
q

quick-keyboard-83126

03/22/2023, 11:11 PM
lima-alpine:/Users/jsoref$ curl "$(./bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948%7C> tail -n 1
|awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/2 200
x-guploader-uploadid: ADPycdtCZKWf_CGfTS9UMNFWe7Ust2QIL5dLccfCBGpim5F3FJoRqUExQqJ6XeG5LqbBhtm79So8RhOBk8XiaSYNyTwQntpvJUR8
expires: Thu, 23 Mar 2023 00:11:00 GMT
date: Wed, 22 Mar 2023 23:11:00 GMT
cache-control: public, max-age=3600
last-modified: Mon, 06 Mar 2023 23:41:48 GMT
etag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
content-type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 9599350
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

  5 9374k    5  520k    0     0  36373      0  0:04:23  0:00:14  0:04:09 32221
doesn't seem to be working
f

fast-garage-66093

03/22/2023, 11:11 PM
Well, it was worth a try
q

quick-keyboard-83126

03/22/2023, 11:11 PM
Sure
f

fast-garage-66093

03/22/2023, 11:12 PM
Unfortunately Nino is sick today; otherwise I would have asked him for ideas
q

quick-keyboard-83126

03/22/2023, 11:12 PM
this has been a problem for months
i have until the end of next week to poke things
otherwise, i'll have to revisit in the second half of april
f

fast-garage-66093

03/22/2023, 11:13 PM
Maybe wait until tomorrow, to see if Nino is better
Ping him and/or me tomorrow as a reminder!
✅ 1
q

quick-keyboard-83126

03/22/2023, 11:48 PM
This is UTM (w/ qemu): 
ubuntu@ubuntu-utm:~$ uname -a
Linux ubuntu-utm 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ubuntu-utm:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy 
ubuntu@ubuntu-utm:~$ curl -4 "$(./google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948%7C> tail -n 1 | awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200
x-guploader-uploadid: ADPycdsXGbECQZIzMFeiBbbaPUgYeKArc7h0bCakwfQPW4LnVDnoxKPs73dr5pmfQMKlcwMuT5JyhK_wofUz61lvVd9e1eoPdXRD
expires: Thu, 23 Mar 2023 00:45:10 GMT
date: Wed, 22 Mar 2023 23:45:10 GMT
cache-control: public, max-age=3600
last-modified: Mon, 06 Mar 2023 23:41:48 GMT
etag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
content-type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 9599350
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

 15 9374k   15 1470k    0     0  14641      0  0:10:55  0:01:42  0:09:13   817
(I tried using UTM w/ Apple's emulator and the networking driver oopsed during the install ...)
f

fast-garage-66093

03/22/2023, 11:50 PM
So you get expected speed?
q

quick-keyboard-83126

03/22/2023, 11:50 PM
It hasn't finished. So it depends on what you're expecting 🙂
This is what should happen (macOS bare): 
jsoref@jsoref-mbp ~ % time curl "$(CLOUDSDK_PYTHON=~/.pyenv/versions/3.10.1/bin/python ./bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95784488c0bbc3e0abff2a6d853109e4a948%7C> tail -n 1 |awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200
x-guploader-uploadid: ADPycdsX26G5uYvSflNNM0FxgC4zhVvq1TsFagqZFQoE38QGV3C3vY7U2YUVeVBbKE9lEd-ofDENOMipnUkvW9K0aXqnrw
expires: Thu, 23 Mar 2023 00:46:40 GMT
date: Wed, 22 Mar 2023 23:46:40 GMT
cache-control: public, max-age=3600
last-modified: Mon, 06 Mar 2023 23:41:48 GMT
etag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
content-type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 9599350
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

100 9374k  100 9374k    0     0  7167k      0  0:00:01  0:00:01 --:--:-- 7194k
curl  -D - -o /dev/null  0.07s user 0.04s system 8% cpu 1.321 total
i.e. a near instant download
From my perspective, this seems to eliminate "lima", although UTM is using QEMU
So, I'm still left w/ QEMU + aarch64 + linux
I suppose I could try UTM w/ amd64 linux to see if that behaves differently...
f

fast-garage-66093

03/22/2023, 11:54 PM
Can you try to switch the networking layer in UTM
I can't remember what is the default, "Emulated VLAN"? Which is probably just QEMU slirp, same as Lima. Try setting e.g. "Bridged" or "Shared" and see if that makes a difference
We have support for this in Lima, but it is only used for ingress, not egress.
q

quick-keyboard-83126

03/22/2023, 11:56 PM
17 9374k 17 1658k 0 0 2738 0 0:58:25 0🔟20 0:48:05 0
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
That's the end of the utm session 🙂
this is what i was using:
I don't think that's the default
f

fast-garage-66093

03/22/2023, 11:57 PM
Yeah, that's already what I wanted you to try, so I guess that's not it either 😞
q

quick-keyboard-83126

03/22/2023, 11:58 PM
the default (shared) doesn't seem to get me network connectivity which is why I switched it (I kinda needed the installer to install ubuntu...)
f

fast-garage-66093

03/23/2023, 12:00 AM
Maybe we do need some packet captures, not for the content, but just to see the traffic flow
q

quick-keyboard-83126

03/23/2023, 12:00 AM
alright. where do you want them captured? in lima?
on macOS?
f

fast-garage-66093

03/23/2023, 12:00 AM
I still somewhat suspect MTU and overflowing buffers because we go through several tunnel layers
q

quick-keyboard-83126

03/23/2023, 12:01 AM
i'm willing to buy that
f

fast-garage-66093

03/23/2023, 12:01 AM
I don't know; let's wait for Nino, who has more experience with this
q

quick-keyboard-83126

03/23/2023, 12:01 AM
i've run into MTU issues occasionally, but I've rarely been able to resolve them 🙂
f

fast-garage-66093

03/23/2023, 12:02 AM
Maybe it needs to be configured on the Gcloud side?
But you are not connecting to a VPC, so I don't know how that would even be possible
q

quick-keyboard-83126

03/23/2023, 12:03 AM
https://issuetracker.google.com/issues/35904605?pli=1 looks fun
f

fast-garage-66093

03/23/2023, 12:06 AM
Normally the connection should negotiate to use the lower value of MTU on either side, but maybe this is not implemented correctly here?
And by "here" I mean the Gcloud side...
q

quick-keyboard-83126

03/23/2023, 12:07 AM
i do have unofficial access to a googler who is fairly network astute, i tossed him a bone, we'll see if he barks at me 🙂
f

fast-garage-66093

03/23/2023, 12:09 AM
Is 
<http://appspot.com|appspot.com>
a Google service? Because that is the site creating the issue (I think)
q

quick-keyboard-83126

03/23/2023, 12:10 AM
yes appspot is google app engine
but, really, in general, most google ips are interchangeable
my naive understanding is that you say hello to a google ip and sni for the service you want and it routes you along
if you're using http3 it might reroute you w/ alt-svc, i suppose, although i haven't watched it do that much and know much less about alt-svc
f

fast-garage-66093

03/23/2023, 12:11 AM
yeah, I just wanted to make sure this is not some 3rd-party hosting service you use for your testing
q

quick-keyboard-83126

03/23/2023, 12:13 AM
oh, ... no appspot is functionally an implementation detail of GCR, https://cloud.google.com/container-registry/docs/access-control
The first image push to a hostname adds the registry host and its storage bucket to a project. For example, the first push to 
<http://gcr.io/my-project|gcr.io/my-project>
adds the 
<http://gcr.io|gcr.io>
registry host to the project with the project ID 
my-project
and creates a storage bucket for the registry. The bucket name has one of the following formats:
• 
<http://artifacts.PROJECT-ID.appspot.com|artifacts.PROJECT-ID.appspot.com>
for images stored on the host 
<http://gcr.io|gcr.io>
• 
<http://STORAGE-REGION.artifacts.PROJECT-ID.appspot.com|STORAGE-REGION.artifacts.PROJECT-ID.appspot.com>
for images stored on other registry hosts
f

fast-garage-66093

03/23/2023, 12:14 AM
I'm going to drop out now and hope by tomorrow that we have some response from your Googler, and also that Nino is back online
q

quick-keyboard-83126

03/23/2023, 12:14 AM
Thanks. I think now's a reasonable time for me to go home too
w

wide-mechanic-33041

03/23/2023, 11:28 AM
yeah usually I will use openssl to test for MTU related issues as well as TLS seems to start throwing handshake errors as well from fragmentation. Long thread, but did the ip link commands work to set it via rdctl shell for a session?
f

fast-garage-66093

03/23/2023, 6:52 PM
@wide-mechanic-33041 afaiu the command worked, but made no difference. I wonder if setting it to 
1000
was too low; I read somewhere that the minimum value for Google Cloud is 
1300
. So maybe the lower value is ignored in MSS negotiations.
w

wide-mechanic-33041

03/23/2023, 7:24 PM
yeah I mean too low probably would impact perceived speed and maybe some stability. I usually end up 1200-1250 with vms on my laptop assuming container, vm, vpn, and then some vxlan upstream. i would likley tcpdump the interface and see if there are a ton of retries or other TCP artifacts. Then work outward as best as possible
q

quick-keyboard-83126

03/23/2023, 7:55 PM
So I should try 1200?
c

calm-sugar-3169

03/23/2023, 7:56 PM
@quick-keyboard-83126 Although I haven’t read all the messages I may have a few suggestions, are you able to use traceroute? If so, you can use the 
--mtu
option to discover the MTU between your network and the destination, please note that 
--mtu
is not always available on all traceroute utilities. Furthermore, I would also play around with the payload size and would capture that packet to see any signs of fragmentation of the packets or even you could look for 
ICMP message-too-big
error messages or 
ICMPv6 packet-too-big
errors (depending on your protocol)
Basically the idea here is when the packet is encapsulated beyond the network MTU, like if the packets go through 
VXLAN
there will be an additional 50 bytes added to the overall size (+ IP header, etc), there usually will be some kind if 
ICMP
messages sent back to the origin to indicated if the packets needs to be resized, etc.
w

wide-mechanic-33041

03/23/2023, 8:15 PM
yeah if you just do an “ip” in alpine it shoudl show the mtu of the interface
q

quick-keyboard-83126

03/23/2023, 8:16 PM
I'm naturally in Alpine. I'm happy to run whatever commands people suggest
The more precise the commands the happier I am
w

wide-mechanic-33041

03/23/2023, 8:16 PM
there are tons of layers with remote workers in the mix so dialing it in can take a few to settle. I found the biggest flag of fragmentation was TLS issues (handshake errors) but any socket could be impacted.
f

fast-garage-66093

03/23/2023, 8:17 PM
For the benefit of those who did not read the whole backlog (hah), Josh already set the MTU to 1000 yesterday and it made no difference: https://rancher-users.slack.com/archives/C0200L1N1MM/p1679526617272239?thread_ts=1679508822.589629&amp;cid=C0200L1N1MM
w

wide-mechanic-33041

03/23/2023, 8:17 PM
and you can always tcpdump eth0 to see if you are seeing other types of TCP issues
q

quick-keyboard-83126

03/23/2023, 8:18 PM
should i be in lima or rancher?
(i don't care, i just prefer precision, and the least amount of thinking on my part -- i'm trying to solve another production problem atm so this has my secondary focus)
w

wide-mechanic-33041

03/23/2023, 8:18 PM
in alpine (rdctl shell) “ip link set dev eth0 mtu 1250” which I think i had a config script on my windows machines
f

fast-garage-66093

03/23/2023, 8:19 PM
I would from now on just test in RD, as we have seen yesterday that the behaviour is the same everywhere (rd shell, lima alpine, lima ubuntu, utm alpine)
✅ 1
👍 1
q

quick-keyboard-83126

03/23/2023, 8:19 PM
lima-rancher-desktop:/Users/jsoref$ ip link set dev eth0 mtu 1250
ip: ioctl 0x8922 failed: Operation not permitted
lima-rancher-desktop:/Users/jsoref$ sudo ip link set dev eth0 mtu 1250
lima-rancher-desktop:/Users/jsoref$
f

fast-garage-66093

03/23/2023, 8:19 PM
sudo
?
w

wide-mechanic-33041

03/23/2023, 8:19 PM
yeah was thinking the same thing?!
q

quick-keyboard-83126

03/23/2023, 8:19 PM
Yeah, i did, see the 3rd line 🙂
👍 1
so, what do you want for a traceroute?
w

wide-mechanic-33041

03/23/2023, 8:20 PM
did something else get installed in alpine that has you in as non-root?
q

quick-keyboard-83126

03/23/2023, 8:20 PM
afaiu, 
rdctl shell
gives me non root by default
ok, what 
sudo apk add
package gets me a 
traceroute
you'll like?
w

wide-mechanic-33041

03/23/2023, 8:21 PM
you had mentioned an endpoint that was underperforming. Target that with the --mtu and see what shows up
q

quick-keyboard-83126

03/23/2023, 8:21 PM
busybox's traceroute isn't mtu friendly
w

wide-mechanic-33041

03/23/2023, 8:22 PM
heh always the case. maybe do a tcpdump on the interface and do some basic curl statements to see if you can spot retries or other TCP errors
f

fast-garage-66093

03/23/2023, 8:23 PM
sudo apk add traceroute
?
lima-rancher-desktop:~# traceroute 2>&1 | grep mtu
  --mtu                       Discover MTU along the path being traced. Implies
q

quick-keyboard-83126

03/23/2023, 8:25 PM
lima-rancher-desktop:/Users/jsoref$ traceroute 2>&1 | grep mtu
lima-rancher-desktop:/Users/jsoref$
lima-rancher-desktop:/Users/jsoref$ traceroute 2>&1 | grep mtu
  --mtu                       Discover MTU along the path being traced. Implies
ok
lima-rancher-desktop:/Users/jsoref$ traceroute --mtu artifacts.gcr-public-test-1
<http://2345.appspot.com|2345.appspot.com>
traceroute to <http://artifacts.gcr-public-test-12345.appspot.com|artifacts.gcr-public-test-12345.appspot.com> <tel:(142.251.32.84|(142.251.32.84>), 30 hops max, 65000 byte packets
 1  host.lima.internal (192.168.5.2)  0.678 ms F=1250  0.459 ms  0.362 ms
 2  * * *
 3  * * *
c

calm-sugar-3169

03/23/2023, 8:29 PM
did it not finish?
is ICMP getting blocked?
q

quick-keyboard-83126

03/23/2023, 8:29 PM
lima-rancher-desktop:/Users/jsoref$ traceroute --mtu artifacts.gcr-public-test-1
<http://2345.appspot.com|2345.appspot.com>
traceroute to <http://artifacts.gcr-public-test-12345.appspot.com|artifacts.gcr-public-test-12345.appspot.com> <tel:(142.251.32.84|(142.251.32.84>), 30 hops max, 65000 byte packets
 1  host.lima.internal (192.168.5.2)  0.678 ms F=1250  0.459 ms  0.362 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * *
c

calm-sugar-3169

03/23/2023, 8:29 PM
ha, ok something is blocking the ICMP
f

fast-garage-66093

03/23/2023, 8:29 PM
Yeah, traceroute doesn't seem to work inside the VM 😞
c

calm-sugar-3169

03/23/2023, 8:30 PM
this might explain why the source fails to adjust the packets sizes which could explain why this whole thing might be failing 😐
q

quick-keyboard-83126

03/23/2023, 8:31 PM
i'll wait for someone to translate (we're fighting GKE in the foreground)
c

calm-sugar-3169

03/23/2023, 8:31 PM
because the entire network fragmentation is driven based on the ICMP messages that is sent back from destination to source, I think
q

quick-keyboard-83126

03/23/2023, 8:34 PM
ok... so what do you folks want me to do? 🙂
w

wide-mechanic-33041

03/23/2023, 8:40 PM
so you confirmed that even w 1250 your performance to the destination is still poor?
q

quick-keyboard-83126

03/23/2023, 8:41 PM
yes.
29  * * *
30  * * *
lima-rancher-desktop:/Users/jsoref$ curl --http1.1 -4 "$(~/gcloud/google-cloud-s
dk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-publi>
<http://c-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95|c-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa95>
784488c0bbc3e0abff2a6d853109e4a948| tail -n 1 |awk '{print $5}')" -D - -o /dev/n
ull
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/1.1 200 OK
X-GUploader-UploadID: ADPycdsf8PjzNktWQt3o6jaexZkbRD_ofKSEYUSdLTmx4xyxu_TXz387MTzS7DLPOdk-sD9euV_K5StA51LY2ICDq8oKag
Expires: Thu, 23 Mar 2023 21:41:06 GMT
Date: Thu, 23 Mar 2023 20:41:06 GMT
Cache-Control: public, max-age=3600
Last-Modified: Mon, 06 Mar 2023 23:41:48 GMT
ETag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
Content-Type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
Accept-Ranges: bytes
Content-Length: 9599350
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

  9 9374k    9  856k    0     0  45934      0  0:03:28  0:00:19  0:03:09 72842
(this transfer should finish nearly instantaneously)
(instead, it will effectively never finish)
w

wide-mechanic-33041

03/23/2023, 8:41 PM
apk add tcpdump
q

quick-keyboard-83126

03/23/2023, 8:42 PM
lima-rancher-desktop:/Users/jsoref$ sudo apk add tcpdump
OK: 470 MiB in 174 packages
w

wide-mechanic-33041

03/23/2023, 8:43 PM
tcpdump -w gcp.pcap -i eth0
✅ 1
q

quick-keyboard-83126

03/23/2023, 8:43 PM
lima-rancher-desktop:/Users/jsoref$ tcpdump -w gcp.pcap -i eth0
tcpdump: eth0: You don't have permission to capture on that device
(socket: Operation not permitted)
lima-rancher-desktop:/Users/jsoref$ sudo tcpdump -w gcp.pcap -i eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
w

wide-mechanic-33041

03/23/2023, 8:43 PM
and while that is running you will need to do your curl in a different session
✅ 1
q

quick-keyboard-83126

03/23/2023, 8:44 PM
ok, killing the curl and then ^c to the tcpdump...
w

wide-mechanic-33041

03/23/2023, 8:44 PM
once you have let curl go for a few seconds you can stop the dump witha ctl+c
q

quick-keyboard-83126

03/23/2023, 8:44 PM
lima-rancher-desktop:/Users/jsoref$ sudo tcpdump -w gcp.pcap -i eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C498 packets captured
503 packets received by filter
0 packets dropped by kernel
w

wide-mechanic-33041

03/23/2023, 8:46 PM
copy out to your host using like the c$ mapping and use like wireshark to open it up. https://www.golinuxcloud.com/packet-fragmentation-wireshark/
q

quick-keyboard-83126

03/23/2023, 8:46 PM
i didn't need to copy it out... because rdctl aliases 
~
into the shell
gcp.pcap
gcp.pcap
w

wide-mechanic-33041

03/23/2023, 8:47 PM
wireshark tends to highlight unexpected things
q

quick-keyboard-83126

03/23/2023, 8:48 PM
well, there are a couple of gray FIN,ACK, but, otherwise i don't see anything particularly exciting
w

wide-mechanic-33041

03/23/2023, 8:48 PM
yup looks pretty clean
q

quick-keyboard-83126

03/23/2023, 8:49 PM
i suppose i could capture a narrow host equivalent for the same traffic, but that'd require a bit more effort
w

wide-mechanic-33041

03/23/2023, 8:49 PM
and this is only w gcp destinations or all flows?
q

quick-keyboard-83126

03/23/2023, 8:49 PM
that tcp dump is the one you asked for
w

wide-mechanic-33041

03/23/2023, 8:49 PM
no i meant the slowness
q

quick-keyboard-83126

03/23/2023, 8:49 PM
but 
docker pull
from docker hub doesn't have this behavior
👍 1
Support has asked me to run 
gsutil perfdiag
, so i'm now running that...
👍 1
lima-alpine:/Users/jsoref$ sudo apk add net‑tools
ERROR: unable to select packages:
  net‑tools (no such package):
    required by: world[net‑tools]
what am i doing wrong?
https://pkgs.alpinelinux.org/contents?file=netstat&amp;path=&amp;name=net-tools&amp;branch=v3.17&amp;repo=main&amp;arch=aarch64
Fwiw, I'm going to restart my rancher ...
w

wide-mechanic-33041

03/23/2023, 9:07 PM
and be good to test those curl statements from other WSL2 distros to try and narrow the cause further
f

fast-garage-66093

03/23/2023, 9:08 PM
@wide-mechanic-33041 Josh is on macOS using M1
q

quick-keyboard-83126

03/23/2023, 9:08 PM
I'm on macOS
we tried ubuntu and alpine
both in rancher desktop and in lima w/o rancher desktop
w

wide-mechanic-33041

03/23/2023, 9:08 PM
hah sorry
q

quick-keyboard-83126

03/23/2023, 9:08 PM
and iirc i reproduced w/ UTM w/ ubuntu as well
f

fast-garage-66093

03/23/2023, 9:08 PM
Works on Intel: 
lima-rancher-desktop:~# apk add net-tools
(1/2) Installing mii-tool (2.10-r0)
(2/2) Installing net-tools (2.10-r0)
Executing busybox-1.35.0-r17.trigger
OK: 326 MiB in 109 packages
w

wide-mechanic-33041

03/23/2023, 9:09 PM
was helping others with WSL2 stuff just before this thread lit up. 😅
q

quick-keyboard-83126

03/23/2023, 9:09 PM
s'ok
w

wide-mechanic-33041

03/23/2023, 9:11 PM
any chance https://techmonitor.ai/technology/cloud/google-cloud-problems-server-latency-us-east could be related? I felt like I had seen something about their ingress the last couple days
q

quick-keyboard-83126

03/23/2023, 9:12 PM
this has been a problem for months
👍 1
f

fast-garage-66093

03/23/2023, 9:12 PM
I don't think it is related. Otherwise access from the host would be slow too.
w

wide-mechanic-33041

03/23/2023, 9:14 PM
yeah agreed. just quite an odd issue
f

fast-garage-66093

03/23/2023, 9:17 PM
I think the fact that ICMP is not working might be a hint. I just don't know if it is used for MSS negotiation or not. Reading Resolve IPv4 Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPsec - Cisco looks like it only uses TCP headers, so should work even when ICMP is blocked.
q

quick-keyboard-83126

03/23/2023, 9:31 PM
so, this is 1.8.1: 
lima-rancher-desktop:/tmp/google-cloud-sdk/bin$ curl "$(~/gcloud/google-cloud-sdk/bin/gsutil signurl -r us -d 20m $service_account_key <gs://artifacts.gcr-public-test-12345.appspot.com/containers/images/sha256:7f65636102fd1f499092cb075baa9>
5784488c0bbc3e0abff2a6d853109e4a948 | tail -n 1 |awk '{print $5}')" -D - -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0HTTP/2 200
x-guploader-uploadid: ADPycdvx73Fnw-tKT0XRFBXC0xG6g4trrvF8nlfTZ1pJHd8iFyjtD9Cjx37gxB9rSdCi1lJYfgs73Z7bp42Qoj2FrA4dF6AWg0bT
expires: Thu, 23 Mar 2023 22:27:20 GMT
date: Thu, 23 Mar 2023 21:27:20 GMT
cache-control: public, max-age=3600
last-modified: Mon, 06 Mar 2023 23:41:48 GMT
etag: "e453dccae0def69a0d0c2552647d58a7"
x-goog-generation: 1678146108297435
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 9599350
content-type: application/octet-stream
x-goog-hash: crc32c=gnP47A==
x-goog-hash: md5=5FPcyuDe9poNDCVSZH1Ypw==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 9599350
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

  8 9374k    8  806k    0     0   4160      0  0:38:27  0:03:18  0:35:09   268
f

fast-garage-66093

03/23/2023, 9:32 PM
Same as before, right? I did not expect the RD version to make any difference here
q

quick-keyboard-83126

03/23/2023, 9:32 PM
right
i did that mostly because it reset my MTU and i didn't want to think about any more variables
now i get to try to figure out this fun 
netstat
/ 
net-tools
thing
f

fast-garage-66093

03/23/2023, 9:34 PM
Not sure why it fails for you. Just tested on M1 and works too: 
$ rd shell sudo -i
lima-rancher-desktop:~# apk add net-tools
WARNING: Ignoring /media/vda/apks: No such file or directory
fetch <https://dl-cdn.alpinelinux.org/alpine/v3.16/main/aarch64/APKINDEX.tar.gz>
fetch <https://dl-cdn.alpinelinux.org/alpine/v3.16/community/aarch64/APKINDEX.tar.gz>
(1/2) Installing mii-tool (2.10-r0)
(2/2) Installing net-tools (2.10-r0)
Executing busybox-1.35.0-r17.trigger
OK: 286 MiB in 107 packages
lima-rancher-desktop:~#
q

quick-keyboard-83126

03/23/2023, 9:34 PM
it worked on rd1.8.1's, just not lima's
really puzzled
f

fast-garage-66093

03/23/2023, 9:36 PM
Worked for me on Lima too: 
$ l shell alpine sudo -i
lima-alpine:~# apk add net-tools
fetch <https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz>
fetch <https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz>
(1/2) Installing mii-tool (2.10-r0)
(2/2) Installing net-tools (2.10-r0)
Executing busybox-1.35.0-r29.trigger
OK: 43 MiB in 86 packages
lima-alpine:~#
q

quick-keyboard-83126

03/23/2023, 9:36 PM
ok
the problem was that some stupid web page gave me a fancy 
en-dash
instead of a 
-
😞 1
🤦‍♂️
6 Views
#rancher-desktopJoin Slack
Powered by