I followed guide <https k3d io v5 4 4 usage registries prefa rancher-users #k3d

Title

polite-engineer-55788

08/10/2022, 2:08 PM

I followed guide https://k3d.io/v5.4.4/usage/registries/#preface-referencing-local-registries to configure the local registry

when I try to build and push the image I see the following error

+ /kaniko/executor --context /drone/src --dockerfile Dockerfile  --customPlatform=linux/amd64 --customPlatform=linux/arm64 --destination $DESTINATION_IMAGE --digest-file /tmp/images/image-digest --insecure --skip-tls-verify\n
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "******": creating push check transport for myregistry.localhost:5001 failed: Get "<https://myregistry.localhost:5001/v2/>": dial tcp 192.168.128.6:5001: connect: connection refused; Get "<http://myregistry.localhost:5001/v2/>": dial tcp 192.168.128.6:5001: connect: connection refused

I have this

registries:
  create:
    name: "${REGISTRY_NAME}"
    host: "0.0.0.0"
    hostPort: "${REGISTRY_PORT}"

REGISTRY_NAME is

myregistry

and port is

any clue ?

wide-garage-9465

08/10/2022, 2:15 PM

Where are you running the build/push relative to the k3d cluster?

polite-engineer-55788

08/10/2022, 2:16 PM

yeah its from a pod in the k3s cluster

apiVersion: <http://k3d.io/v1alpha4|k3d.io/v1alpha4>
kind: Simple
metadata:
  name: dag
servers: 1
agents: 2
image: rancher/k3s:v1.24.3-k3s1
ports:
  # Drone CI
  - port: 127.0.0.1:30980:30980
    nodeFilters:
      - agent:*
  # Gitea
  - port: 127.0.0.1:30950:30950
    nodeFilters:
      - agent:*
  # Argo CD
  - port: 127.0.0.1:30080:30080
    nodeFilters:
      - agent:*
registries:
  create:
    name: "myregistry.localhost"
    host: "0.0.0.0"
    hostPort: "5001"
    volumes:
      - "/Users/kameshs/MyLabs/gitops/dag/.k3s/registry:/var/lib/registry"

thats my config

wide-garage-9465

08/10/2022, 2:19 PM

I think it should be

k3d-myregistry.localhost:5001

, shouldn't it? There are some notes about it in the docs. The

k3d-

prefix is added automatically during creation.

polite-engineer-55788

08/10/2022, 4:50 PM

you mean while creating using YAML config ? I thought it was while creating via CLI

let me try it

seeing the same issue with

k3d

prefix as well from a pod within the cluster

creating push check transport for k3d-myregistry.localhost:5001 failed: Get "<https://k3d-myregistry.localhost:5001/v2/>": dial tcp 192.168.144.2:5001: connect: connection refused; Get "<http://k3d-myregistry.localhost:5001/v2/>": dial tcp 192.168.144.2:5001: connect: connection refused

wide-garage-9465

08/10/2022, 5:23 PM

That IP it resolves to looks weird given that it should be a docker container IP 🤔 Can you resolve the name in any other pod just using e.g. ping or nslookup? And can you please share the output of

kubectl get cm -n kube-system coredns -o yaml

polite-engineer-55788

08/10/2022, 5:46 PM

I am able to hit

<https://k3d-myregistry.localhost:5001/v2/>

from the host

Corefile

yaml
 Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    import /etc/coredns/custom/*.server
  NodeHosts: |
    192.168.65.2 host.k3d.internal
    192.168.144.2 k3d-myregistry.localhost
    192.168.144.7 k3d-dag-serverlb
    192.168.144.6 k3d-dag-agent-0
    192.168.144.3 k3d-dag-tools
    192.168.144.4 k3d-dag-server-0
    192.168.144.5 k3d-dag-agent-1

from one of the test pods when I do

ns-lookup

I get like

/apps # nslookup k3d-myregistry.localhost
Server:         10.43.0.10
Address:        10.43.0.10:53


Name:   k3d-myregistry.localhost
Address: 192.168.144.6

but doing curl fails

/apps # curl 192.168.144.6:5001/v2
curl: (7) Failed to connect to 192.168.144.6 port 5001 after 0 ms: Connection refused
/apps # curl <http://192.168.144.6:5001/v2>
curl: (7) Failed to connect to 192.168.144.6 port 5001 after 0 ms: Connection refused

the cluster was created using the config

yaml
apiVersion: <http://k3d.io/v1alpha4|k3d.io/v1alpha4>
kind: Simple
metadata:
  name: dag
servers: 1
agents: 2
image: rancher/k3s:v1.24.3-k3s1
ports:
  # Drone CI
  - port: 127.0.0.1:30980:30980
    nodeFilters:
      - agent:*
  # Gitea
  - port: 127.0.0.1:30950:30950
    nodeFilters:
      - agent:*
  # Argo CD
  - port: 127.0.0.1:30080:30080
    nodeFilters:
      - agent:*
registries:
  create:
    name: "k3d-myregistry.localhost"
    host: "0.0.0.0"
    hostPort: "5001"
    volumes:
      - "/Users/kameshs/MyLabs/gitops/dag/.k3s/registry:/var/lib/registry"
  config: |
    mirrors:
      "<http://docker.io|docker.io>": 
         endpoint:
           - <https://registry-1.docker.io>

and node

Corefile

looks like

Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    import /etc/coredns/custom/*.server
  NodeHosts: |
    192.168.65.2 host.k3d.internal
    192.168.144.4 k3d-dag-agent-1
    192.168.144.2 k3d-dag-tools
    192.168.144.7 k3d-dag-serverlb
    192.168.144.6 k3d-myregistry.localhost
    192.168.144.3 k3d-dag-server-0
    192.168.144.5 k3d-dag-agent-0

wide-garage-9465

08/10/2022, 6:24 PM

Alright, I guess it's best if you compile all this information into a GitHub issue, ok?

🙌🏽 1

polite-engineer-55788

08/10/2022, 11:34 PM

@wide-garage-9465 I have opened a discussion on GitHub for this https://github.com/k3d-io/k3d/discussions/1124

@wide-garage-9465 any clues

wide-garage-9465

08/12/2022, 6:48 AM

Just commented. Try port 5000 from within the cluster 👍

👀 1

#k3d Join Slack