So I d like to use `docker buildx build` to chain build depe rancher-users #rancher-desktop

Title

# rancher-desktop

quick-keyboard-83126

06/29/2022, 8:49 PM

So, I'd like to use

docker buildx build

to chain build dependencies, but apparently in order to do that, i need to use

--push

, but I don't really want to publish to a remote registry, so I tried using:

Copy code

docker run -d -p 5001:5000 --restart=always --name registry registry:2

(port 5000 is in use by

Server: AirTunes/615.12.1

) But (once I pick my non localhost ip address...), this fails miserably with:

Copy code

error: failed to solve: failed to do request: Head "https://<my-non-localhost-ip>:5001/v2/foo/blobs/sha256:<random-sha>": http: server gave HTTP response to HTTPS client

I understand that

registry

speaks

http

out of the box and

docker

wants to speak

https

. I get that there's a file I could change (but, I suspect that I'd have to change it in my.

lima vm

, and that doing this would be arguably very dangerous). Anyone have suggestions?

fast-garage-66093

06/29/2022, 9:07 PM

You either have to configure it as an "insecure registry" or provide certs for your registry (even self-signed work if you add your CA to you local keychain)

quick-keyboard-83126

06/29/2022, 9:07 PM

This whole thing is really suboptimal

local keychain = macOS?

as opposed to lima?

fast-garage-66093

06/29/2022, 9:08 PM

Yes, Rancher Desktop will automatically import all CA certs into the VM on startup

quick-keyboard-83126

06/29/2022, 9:08 PM

It'd be really nice if there was a: ☑️ Run local docker registry

👍 1

fast-garage-66093

06/29/2022, 9:09 PM

Yeah, I think one day we'll add this, but not right now

BTW, from my notes testing an insecure registry:

Copy code

lima-rancher-desktop:~# tail -1 /etc/conf.d/docker
DOCKER_OPTS="--insecure-registry=insecure.home:80"

You need to restart the docker service after making the change

quick-keyboard-83126

06/29/2022, 9:11 PM

This is so very very very messy

fast-garage-66093

06/29/2022, 9:11 PM

This is a docker requirement; only localhost based registry will work without SSL; everything else must be declared as an insecure registry first

quick-keyboard-83126

06/29/2022, 9:12 PM

All I really want is to a way to build a multiplatform image and then build another multiplatform image that depends on that first one

fast-garage-66093

06/29/2022, 9:12 PM

Adding certs to a registry is pretty trivial; no reason not to do it

quick-keyboard-83126

06/29/2022, 9:12 PM

I shouldn't have to set up a registry at all

👍 1

I really really really really don't want to run a registry.

If it was a vaguely simple task to workaround a really stupid wart, I could tolerate doing it. But this far exceeds that.

fast-garage-66093

06/29/2022, 9:13 PM

I don't understand why you have to; do you have a repro case you can share (hopefully somewhat limited in scope)?

quick-keyboard-83126

06/29/2022, 9:13 PM

Yeah, gimme a few bits

Copy code

(cd foo; cat Dockerfile; docker buildx build --platform linux/arm64,linux/amd64 -t foo .); (cd bar; cat Dockerfile; docker buildx build --platform linux/arm64,linux/amd64 -t bar .)
from busybox

RUN echo 'hello' > world
WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.2s (8/8) FINISHED                                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 76B                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => [linux/amd64 internal] load metadata for <http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>                                                                                                          0.2s
 => [linux/arm64 internal] load metadata for <http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>                                                                                                          0.2s
 => [linux/arm64 1/2] FROM <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                           0.0s
 => => resolve <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                                       0.0s
 => [linux/amd64 1/2] FROM <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                           0.0s
 => => resolve <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                                       0.0s
 => CACHED [linux/arm64 2/2] RUN echo 'hello' > world                                                                                                                                  0.0s
 => CACHED [linux/amd64 2/2] RUN echo 'hello' > world                                                                                                                                  0.0s
from foo

RUN echo 'cruel' >> world
WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.3s (4/4) FINISHED                                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 73B                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => CANCELED [linux/arm64 internal] load metadata for <http://docker.io/library/foo:latest|docker.io/library/foo:latest>                                                                                                     0.3s
 => ERROR [linux/amd64 internal] load metadata for <http://docker.io/library/foo:latest|docker.io/library/foo:latest>                                                                                                        0.3s
------
 > [linux/amd64 internal] load metadata for <http://docker.io/library/foo:latest|docker.io/library/foo:latest>:
------
Dockerfile:1
--------------------
   1 | >>> from foo
   2 |     
   3 |     RUN echo 'cruel' >> world
--------------------
error: failed to solve: foo: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Copy code

cat foo/Dockerfile; echo '----'; cat bar/Dockerfile
from busybox

RUN echo 'hello' > world
----
from foo

RUN echo 'cruel' >> world

For comparison:

Copy code

(cd foo; cat Dockerfile; docker build -t foo .); (cd bar; cat Dockerfile; docker build -t bar .)            
from busybox

RUN echo 'hello' > world
Sending build context to Docker daemon  2.048kB
Step 1/2 : from busybox
 ---> 3c277069c6ae
Step 2/2 : RUN echo 'hello' > world
 ---> Using cache
 ---> 68edd097ffa9
Successfully built 68edd097ffa9
Successfully tagged foo:latest
from foo

RUN echo 'cruel' >> world
Sending build context to Docker daemon  2.048kB
Step 1/2 : from foo
 ---> 68edd097ffa9
Step 2/2 : RUN echo 'cruel' >> world
 ---> Running in c282b4b1c190
Removing intermediate container c282b4b1c190
 ---> 65f0f7d6e0b9
Successfully built 65f0f7d6e0b9
Successfully tagged bar:latest

I can "fix" this by not using

-t foo

and

-t bar

, but instead using

-t real.registry/path/foo

and

-t real.registry/path/bar

and updating my

from foo

bar/Dockerfile

from real.registry/path/foo

fast-garage-66093

06/29/2022, 9:22 PM

Yeah, it is a limitation of the exporter:

Copy code

$ docker buildx build --platform linux/arm64,linux/amd64 -t foo . --load
[+] Building 0.0s (0/0)
error: docker exporter does not currently support exporting manifest lists

quick-keyboard-83126

06/29/2022, 9:22 PM

Yeah, I know, I ran that 🙂

It is not a remotely helpful message.

fast-garage-66093

06/29/2022, 9:25 PM

Looks like local registry is the only option; it is a limitation of moby: Building images for multi-arch with --load parameter fails · Issue #59 · docker/buildx

quick-keyboard-83126

06/29/2022, 9:26 PM

local registry = feature request to rancher-desktop to "magic make it work" 🙂

I guess I can file an RFE for it.

fast-garage-66093

06/29/2022, 9:26 PM

Sure, please do, but I don't understand why you can't use the local registry on

localhost

If you don't want to configure it as "insecure"

Copy code

$ docker run -d -p 5001:5000 --restart=always --name registry registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
2408cc74d12b: Pull complete
ea60b727a1ce: Pull complete
c87369050336: Pull complete
e69d20d3dd20: Pull complete
fc30d7061437: Pull complete
Digest: sha256:bedef0f1d248508fe0a16d2cacea1d2e68e899b2220e2258f1b604e1f327d475
Status: Downloaded newer image for registry:2
cda0f8aad69d1a53870bb9f285e3c4a10765e96c233ba277f45f3111ed3de405

$ docker tag registry:2 localhost:5001/registry:2

$ docker push localhost:5001/registry:2
The push refers to repository [localhost:5001/registry]
a1d94f4a4c89: Pushed
1d80677cd085: Pushed
58548fced558: Pushed
79f02b258438: Pushed
24302eb7d908: Pushed
2: digest: sha256:affdb68e24c93262ee98a5bdc371e8f631568e9b4401375e7a0ecc705ac526b2 size: 1363

quick-keyboard-83126

06/29/2022, 9:30 PM

Because it didn't work? 🙂

Copy code

(cd foo; cat Dockerfile; docker buildx build --platform linux/arm64,linux/amd64 -t 127.0.0.1:5001/foo .); (cd bar; cat Dockerfile; docker buildx build --platform linux/arm64,linux/amd64 -t bar .); curl -v 127.0.0.1:5001/foo:latest

from busybox

RUN echo 'hello' > world
WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.1s (8/8) FINISHED                                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 76B                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => [linux/amd64 internal] load metadata for <http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>                                                                                                          0.1s
 => [linux/arm64 internal] load metadata for <http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>                                                                                                          0.1s
 => [linux/amd64 1/2] FROM <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                           0.0s
 => => resolve <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                                       0.0s
 => [linux/arm64 1/2] FROM <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                           0.0s
 => => resolve <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                                       0.0s
 => CACHED [linux/arm64 2/2] RUN echo 'hello' > world                                                                                                                                  0.0s
 => CACHED [linux/amd64 2/2] RUN echo 'hello' > world                                                                                                                                  0.0s
from 127.0.0.1:5001/foo

RUN echo 'cruel' >> world
WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.0s (4/4) FINISHED                                                                                                                                                            
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 88B                                                                                                                                                    0.0s
 => ERROR [linux/amd64 internal] load metadata for 127.0.0.1:5001/foo:latest                                                                                                           0.0s
 => CANCELED [linux/arm64 internal] load metadata for 127.0.0.1:5001/foo:latest                                                                                                        0.0s
------
 > [linux/amd64 internal] load metadata for 127.0.0.1:5001/foo:latest:
------
Dockerfile:1
--------------------
   1 | >>> from 127.0.0.1:5001/foo
   2 |     
   3 |     RUN echo 'cruel' >> world
--------------------
error: failed to solve: 127.0.0.1:5001/foo: failed to do request: Head "<http://127.0.0.1:5001/v2/foo/manifests/latest>": dial tcp 127.0.0.1:5001: connect: connection refused
*   Trying 127.0.0.1:5001...
* Connected to 127.0.0.1 (127.0.0.1) port 5001 (#0)
> GET /foo:latest HTTP/1.1
> Host: 127.0.0.1:5001
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Wed, 29 Jun 2022 21:29:36 GMT
< Content-Length: 19
< 
404 page not found
* Connection #0 to host 127.0.0.1 left intact

I think for it to work, the registry would have to run in lima and not on my mac

fast-garage-66093

06/29/2022, 9:31 PM

Not sure what you mean, it does run in lima; how would it run on your mac?

Let me try your example; I didn't see you push the foo image to the registry though (use

--push

instead of

--load

)

quick-keyboard-83126

06/29/2022, 9:33 PM

Copy code

(cd foo; cat Dockerfile; docker buildx build --platform linux/arm64,linux/amd64 -t 127.0.0.1:5001/foo . --push); (cd bar; cat Dockerfile; docker buildx build --platform linux/arm64,linux/amd64 -t bar .); curl -v 127.0.0.1:5001/foo:latest
from busybox

RUN echo 'hello' > world
[+] Building 0.3s (9/9) FINISHED                                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 76B                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => [linux/amd64 internal] load metadata for <http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>                                                                                                          0.2s
 => [linux/arm64 internal] load metadata for <http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>                                                                                                          0.2s
 => [linux/amd64 1/2] FROM <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                           0.0s
 => => resolve <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                                       0.0s
 => [linux/arm64 1/2] FROM <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                           0.0s
 => => resolve <http://docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83|docker.io/library/busybox@sha256:3614ca5eacf0a3a1bcc361c939202a974b4902b9334ff36eb29ffe9011aaad83>                                                                       0.0s
 => CACHED [linux/amd64 2/2] RUN echo 'hello' > world                                                                                                                                  0.0s
 => CACHED [linux/arm64 2/2] RUN echo 'hello' > world                                                                                                                                  0.0s
 => ERROR exporting to image                                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                                                0.0s
 => => exporting manifest sha256:0d067956ce52bc021a48a4ccd287c3829f4b0eca79a5fa419a35f82d634e25e0                                                                                      0.0s
 => => exporting config sha256:cec764602494d167dbf293bccaf36a0880332bf4037ed31ad81fed0505f6ecb3                                                                                        0.0s
 => => exporting manifest sha256:8cc4b99868c5c81d709fe248d69f899ba04267d340f13424bcc10e8704857770                                                                                      0.0s
 => => exporting config sha256:3efad4c0fdf3cf85b6ef37fea7eb9092a5ba9ad72e0bb2ff55eb9f33534527d9                                                                                        0.0s
 => => exporting manifest list sha256:29131e9a7b5a8cbe1a5e93e8427332a2ade54eea915d97fbb3dfeb445102e866                                                                                 0.0s
 => => pushing layers                                                                                                                                                                  0.0s
------
 > exporting to image:
------
error: failed to solve: failed to do request: Head "<http://127.0.0.1:5001/v2/foo/blobs/sha256:cec764602494d167dbf293bccaf36a0880332bf4037ed31ad81fed0505f6ecb3>": dial tcp 127.0.0.1:5001: connect: connection refused
from 127.0.0.1:5001/foo

RUN echo 'cruel' >> world
WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.0s (4/4) FINISHED                                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 88B                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => ERROR [linux/amd64 internal] load metadata for 127.0.0.1:5001/foo:latest                                                                                                           0.0s
 => CANCELED [linux/arm64 internal] load metadata for 127.0.0.1:5001/foo:latest                                                                                                        0.0s
------
 > [linux/amd64 internal] load metadata for 127.0.0.1:5001/foo:latest:
------
Dockerfile:1
--------------------
   1 | >>> from 127.0.0.1:5001/foo
   2 |     
   3 |     RUN echo 'cruel' >> world
--------------------
error: failed to solve: 127.0.0.1:5001/foo: failed to do request: Head "<http://127.0.0.1:5001/v2/foo/manifests/latest>": dial tcp 127.0.0.1:5001: connect: connection refused
*   Trying 127.0.0.1:5001...
* Connected to 127.0.0.1 (127.0.0.1) port 5001 (#0)
> GET /foo:latest HTTP/1.1
> Host: 127.0.0.1:5001
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Wed, 29 Jun 2022 21:32:47 GMT
< Content-Length: 19
< 
404 page not found
* Connection #0 to host 127.0.0.1 left intact

Fwiw, here's the command I used "on my mac" to start the registry:

Copy code

% docker run -d -p 5001:5000 --restart=always --name registry registry:2

da7c130ba1956c9c191c8d5957459b19d34cdc33900377381cbb7208222f4c63

fast-garage-66093

06/29/2022, 9:34 PM

Yeah, it doesn't work because buildkit runs inside a container, so

localhost

refers to the container and not the VM 😞

💡 1

quick-keyboard-83126

06/29/2022, 9:34 PM

Anyway, I can't otherwise explain why it wouldn't work, you're right they're all running in lima

Of course.

Alright, lemme see if I can write this up

Am I right that if I was on Linux and didn't have rancher-desktop or docker-desktop, this series of commands would "magically work"?

Or does this never work w/o a registry?

fast-garage-66093

06/29/2022, 9:38 PM

I think it would never work without a registry because you cannot load "foreign" images into Moby; it will always only load the layers that match the current arch

quick-keyboard-83126

06/29/2022, 9:39 PM

Hmm. So, basically people never do this?

It seems strange to me that there couldn't be two "native" image sets, one for "host native" and one for "other platform -- which from its own perspective was its own native"

(roughly running a

moby

linux/amd64

and talking to it when doing

build... --platform linux/amd64

and running a

moby

linux/arm64

and talking to it when doing

build... -platform linux/arm64

...)

fast-garage-66093

06/29/2022, 9:42 PM

Well, it isn't implemented in buildkit afaict

quick-keyboard-83126

06/29/2022, 9:42 PM

Certainly willing to believe that 🙂

I'm pretty sure I don't want to know what buildkit is doing instead...

fast-garage-66093

06/29/2022, 9:42 PM

So either defining an insecure registry, or adding certs to your registry seem to be the only ways forward

quick-keyboard-83126

06/29/2022, 9:43 PM

"file a bug asking for rancher-desktop to do it for me" seems like a very good way forward for me 🙂

fast-garage-66093

06/29/2022, 9:43 PM

BTW, I find FiloSottile/mkcert: A simple zero-config tool to make locally trusted development certificates with any names you'd like. useful for quickly generating the certs and adding them to your registry

quick-keyboard-83126

06/29/2022, 9:44 PM

I'm fully capable of making certs. But there are way too many steps for me to ask a bunch of my coworkers to do. Especially for something they should need to deal w/ once every 9-30 months.

And they're way too intricate afaict, since some of them involve jumping into lima and restarting things

(or worse, not lima, in case we ever run rancher-desktop on Windows, which we may)

fast-garage-66093

06/29/2022, 9:45 PM

Oh, I thought this was just for yourself

quick-keyboard-83126

06/29/2022, 9:45 PM

No, I represent a dozen folks 🙂

fast-garage-66093

06/29/2022, 9:46 PM

Sure, please file an issue. Adding a registry is one thing though, but then adding a UI on top is something else...

quick-keyboard-83126

06/29/2022, 9:46 PM

I've already built the image we actually care about (or rather dragged someone else through doing it, using our live remote https repository)

I'm happy to design the UI. I'm half doing that for a number of public products 🙂

Heck, at times I might even make the UI code, although I do try to avoid it 🙂

fast-garage-66093

06/29/2022, 9:47 PM

There is SUSE/Portus: Authorization service and frontend for Docker registry (v2) but it seems to be dead

😢 1

So, create the Github issue, and get other people to "upvote" it (not just your coworkers 🙂)

quick-keyboard-83126

06/29/2022, 9:52 PM

Yeah, we don't have the energy to cheat like that 🙂

fast-garage-66093

06/29/2022, 9:54 PM

I think a registry will also be useful if you are using a multi-node cluster because you can no longer build the images on the node; the image must be loadable from multiple nodes.

Of course RD doesn't support multi-node clusters directly (yet?), but you can get them with

k3d

quick-keyboard-83126

06/29/2022, 9:55 PM

I'll leave that to you to write. I've used

k3d

, but don't have the energy to reason out the modifications to this text.

fast-garage-66093

06/29/2022, 9:55 PM

No, just write up your own use case; this was just brainstorming

Another thing I've been thinking about before: if we do add registry support, I would like it to store the images on the host, and not inside the VM, so I can recreate the VM without losing the images. Not sure if that is important or not; may be more something to think about for air-gapped installations

👍 1

(don't add that to your issue either; still brainstorming)

quick-keyboard-83126

06/29/2022, 9:58 PM

That's just a matter of the docker run thing giving a volume for storage to the registry, no?

fast-garage-66093

06/29/2022, 9:59 PM

Not really, as all the volumes are still stored inside the VM

This may be another underdeveloped area of Rancher Desktop 🙂

quick-keyboard-83126

06/29/2022, 10:00 PM

If rancher-desktop does a docker run, it doesn't map a volume into the vm?

fast-garage-66093

06/29/2022, 10:01 PM

No volumes are still blobs of data inside the VM, or are you talking about volume mounts (bind mounts), which are indeed bound from the host

But volumes are also a docker concept to store data, and those have typically much better performance than bind-mounts from the host

Use volumes | Docker Documentation

So "docker volumes" are stored under

/var/lib/docker/volumes/

inside the VM...

quick-keyboard-83126

06/29/2022, 10:07 PM

Ah, sorry, I clearly meant

volume mounts

I guess I do know that volumes technically exist as things, but, we're a kubernetes house, so I don't care that docker technically supports such things most of the time 🙂

fast-garage-66093

06/29/2022, 10:08 PM

Ok, yeah, they live on the host, but have pretty poor performance

quick-keyboard-83126

06/29/2022, 10:08 PM

Yeah, I'm pretty sure I don't care about perf for this

I'd assume most of the time is spent building images as opposed to the final store or the initial load

But, if someone wants to figure out a way to optimize that problem, they're welcome to 🙂.

fast-garage-66093

06/29/2022, 10:09 PM

Well, we'll still need to deal with the size of the registry data, and ways of pruning it etc. Nothing is ever easy.

quick-keyboard-83126

06/29/2022, 10:10 PM

Sure, but isn't that roughly equivalent to the problem exposed by the

images

pane?

fast-garage-66093

06/29/2022, 10:10 PM

People already complain that the VM volume grows to 100GB because they are running on Laptops with puny disks

quick-keyboard-83126

06/29/2022, 10:10 PM

See me adding tasks to https://rancher-users.slack.com/archives/C0200L1N1MM/p1656537831543789

Yeah, I tried to get a 4TB recently

Instead, I have a 1TB mbp

and it's 3/4 full 🙂

fast-garage-66093

06/29/2022, 10:11 PM

The images are stored inside the VM, so are also bound by the 100GB max. You will run out of memory, and then have to clean them. But you can also use

docker prune

I think Guna has already added your tasks to the epic

quick-keyboard-83126

06/29/2022, 10:12 PM

Yeah, probably. Basically, I'm saying I'm aware that there needs to be a related UI to surface this stuff, especially for pruning 🙂

fast-garage-66093

06/29/2022, 10:12 PM

Anyways, my point is that pruning a registry is harder than pruning the images in the runtime because there is no simple CLI for it

quick-keyboard-83126

06/29/2022, 10:13 PM

boggles

fast-garage-66093

06/29/2022, 10:13 PM

Adding the registry and making it accessible is the easy part...

quick-keyboard-83126

06/29/2022, 10:13 PM

ok, I think I'm missing a lot here. ELI5. Can a registry not be treated as a flat set of images just like the images view?

fast-garage-66093

06/29/2022, 10:15 PM

Yeah, but images consists of manifests, digests, layers, etc, which are separate "blobs". You don't have one file per image

quick-keyboard-83126

06/29/2022, 10:16 PM

Right

fast-garage-66093

06/29/2022, 10:16 PM

So if your images share layers, there is only a single copy of that layer in the registry

quick-keyboard-83126

06/29/2022, 10:16 PM

But is the rest just a matter of having a garbage collector?

or rather, don't you have the same problem w/ the

images

feature today?

fast-garage-66093

06/29/2022, 10:16 PM

Yes

quick-keyboard-83126

06/29/2022, 10:16 PM

don't images share layers?

fast-garage-66093

06/29/2022, 10:17 PM

Yes, but you can do this:

Copy code

$ docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] ^C

And you can list the images, and delete the ones you no longer need.

quick-keyboard-83126

06/29/2022, 10:19 PM

Ah. Well, a button in the registry page that says

prune dangling images

or a behavior that just automatically garbage collects whenever anyone removes a tag from the regsitry...

fast-garage-66093

06/29/2022, 10:19 PM

You need 3rd-party tools like

reg

to even list the images in a registry

quick-keyboard-83126

06/29/2022, 10:19 PM

I guess I'm missing a detail

I can publish an untagged sha to a registry

so it's really only garbage when i actively delete the sha (having already deleted all the tags) and when no other images use its layers.

but it still feels like one could garbage collect any time someone performs a delete operation.

fast-garage-66093

06/29/2022, 10:21 PM

Yeah, I don't know; nowadays you can store other things besides images in registries too (like helm charts), so I don't know how it works

quick-keyboard-83126

06/29/2022, 10:21 PM

(and a narrow gc at that, since you're only looking to see if the thing you deleted is the last reference for an object)

we run

nexus

which can host all of these fun blobs

(and we have gcr.io, and technically the google replacement for it...)

(Artifact Registry)

*.pkg.dev

fast-garage-66093

06/29/2022, 10:23 PM

ok, this is ballooning out of scope now; maybe running a registry inside the VM, and destroy all the data when you destroy the VM is the way to start

quick-keyboard-83126

06/29/2022, 10:24 PM

https://github.com/rancher-sandbox/rancher-desktop/issues/2475

👍 1

fast-garage-66093

06/29/2022, 10:24 PM

Conceptually it would be just an image cache for local images

We could still use it as an insecure registry on

192.168.5.15

; we just need to declare it as insecure...

quick-keyboard-83126

06/29/2022, 10:27 PM

Ok -- I'll leave fixing the issue to you. It was too many things to keep in my head 🙂

fast-garage-66093

06/29/2022, 10:27 PM

ok, thanks for creating the issue; we'll continue this at a later time 🙂

✅ 1

quick-keyboard-83126

06/30/2022, 12:02 AM

looks like https://github.com/docker/buildx/issues/777#issuecomment-928000274 has info on how to set up insecure registry w/ the buildx builder...

Copy code

cat > buildkit-config.toml <<EOF
# Custom registry.
[registry."192.168.0.2:5050"]
  http = true
  insecure = true
EOF
docker buildx create --use --config buildkit-config.toml

Not sure if that's compatible w/ the stuff I'm already doing, but.

10 Views