Hi all, I try to switch from Kind to k3d. But I fail in the very beginning because I am using flux to deploy my stuff. If I deploy the following manifest:

apiVersion: <http://source.toolkit.fluxcd.io/v1beta1|source.toolkit.fluxcd.io/v1beta1>

kind: HelmRepository metadata: name: vault spec: interval: 5m url: https://helm.releases.hashicorp.com I got the following error from flux:

❯ kg <http://helmrepositories.source.toolkit.fluxcd.io|helmrepositories.source.toolkit.fluxcd.io> vault

NAME URL AGE READY STATUS vault https://helm.releases.hashicorp.com 10m False failed to fetch Helm repository index: failed to cache index to temporary file: Get "https://helm.releases.hashicorp.com/index.yaml": dial tcp: lookup helm.releases.hashicorp.com on 10.43.0.10:53: server misbehaving on the Kind / rke / minikube it works perfect. I have no glue what's going wrong there in k3d. Can anybody help me please in this issue Thanks a lot Klaus

Hi there 👋 Can you provide some more details please? Which versions of k3d and K3s are you using? As this seems to be an issue with name resolution, potentially coming from CoreDNS, can you check the CoreDNS pod in kube-system? I just tried it and it worked without issues on k3d v5.4.2 🤔

$ k3d cluster create                            
INFO[0000] Loadbalancer image set from env var $K3D_IMAGE_LOADBALANCER: <http://ghcr.io/k3d-io/k3d-proxy:5.4.2|ghcr.io/k3d-io/k3d-proxy:5.4.2> 
INFO[0000] Loadbalancer image set from env var $K3D_IMAGE_LOADBALANCER: <http://ghcr.io/k3d-io/k3d-proxy:5.4.2|ghcr.io/k3d-io/k3d-proxy:5.4.2> 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created image volume k3d-k3s-default-images  
INFO[0000] Starting new tools node...                   
INFO[0000] Tools image set from env var $K3D_IMAGE_TOOLS: <http://ghcr.io/k3d-io/k3d-tools:5.4.2|ghcr.io/k3d-io/k3d-tools:5.4.2> 
INFO[0000] Starting Node 'k3d-k3s-default-tools'        
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] HostIP: using network gateway 172.19.0.1 address 
INFO[0001] Starting cluster 'k3s-default'               
INFO[0001] Starting servers...                          
INFO[0001] Starting Node 'k3d-k3s-default-server-0'     
INFO[0005] All agents already running.                  
INFO[0005] Starting helpers...                          
INFO[0005] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0012] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
INFO[0014] Cluster 'k3s-default' created successfully!  
INFO[0014] You can now use it like this:                
kubectl cluster-info

$ flux install                                                     
✚ generating manifests
✔ manifests build completed
► installing components in flux-system namespace
CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io created
CustomResourceDefinition/buckets.source.toolkit.fluxcd.io created
CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io created
CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io created
CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io created
CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io created
CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io created
CustomResourceDefinition/providers.notification.toolkit.fluxcd.io created
CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io created
Namespace/flux-system created
ServiceAccount/flux-system/helm-controller created
ServiceAccount/flux-system/kustomize-controller created
ServiceAccount/flux-system/notification-controller created
ServiceAccount/flux-system/source-controller created
ClusterRole/crd-controller-flux-system created
ClusterRoleBinding/cluster-reconciler-flux-system created
ClusterRoleBinding/crd-controller-flux-system created
Service/flux-system/notification-controller created
Service/flux-system/source-controller created
Service/flux-system/webhook-receiver created
Deployment/flux-system/helm-controller created
Deployment/flux-system/kustomize-controller created
Deployment/flux-system/notification-controller created
Deployment/flux-system/source-controller created
NetworkPolicy/flux-system/allow-egress created
NetworkPolicy/flux-system/allow-scraping created
NetworkPolicy/flux-system/allow-webhooks created
◎ verifying installation
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ install finished

$ kgp -n flux-system
NAME                                      READY   STATUS    RESTARTS   AGE
helm-controller-7cdc7874f8-r5vr6          1/1     Running   0          2m44s
notification-controller-57cd4f4df-2trzj   1/1     Running   0          2m44s
source-controller-57cf5b9885-f6k2j        1/1     Running   0          2m44s
kustomize-controller-88567bd88-9796d      1/1     Running   0          2m44s

$ cat<<EOF | kubectl apply -f -
pipe heredoc> apiVersion: <http://source.toolkit.fluxcd.io/v1beta1|source.toolkit.fluxcd.io/v1beta1>
pipe heredoc> kind: HelmRepository     
pipe heredoc> metadata:   
pipe heredoc>   name: vault
pipe heredoc> spec:                         
pipe heredoc>   interval: 5m
pipe heredoc>   url:  <https://helm.releases.hashicorp.com>
pipe heredoc> EOF
<http://helmrepository.source.toolkit.fluxcd.io/vault|helmrepository.source.toolkit.fluxcd.io/vault> created

$ k get <http://helmrepositories.source.toolkit.fluxcd.io|helmrepositories.source.toolkit.fluxcd.io>   
NAME    URL                                   AGE   READY   STATUS
vault   <https://helm.releases.hashicorp.com>   10s   True    stored artifact for revision 'fd98f29559cbb6bfa028e9ce533d6d5915aa3aa7c6a03095b95fcc2f14a53a30'

Hi, sure:

❯ k3d version
k3d version v5.4.3
k3s version v1.23.6-k3s1 (default)

k3d cluster create demo
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-demo'                   
INFO[0000] Created image volume k3d-demo-images         
INFO[0000] Starting new tools node...                   
INFO[0000] Starting Node 'k3d-demo-tools'               
INFO[0001] Creating node 'k3d-demo-server-0'            
INFO[0001] Creating LoadBalancer 'k3d-demo-serverlb'    
INFO[0002] Using the k3d-tools node to gather environment information 
INFO[0002] HostIP: using network gateway 172.23.0.1 address 
INFO[0002] Starting cluster 'demo'                      
INFO[0002] Starting servers...                          
INFO[0003] Starting Node 'k3d-demo-server-0'            
INFO[0011] All agents already running.                  
INFO[0011] Starting helpers...                          
INFO[0012] Starting Node 'k3d-demo-serverlb'            
INFO[0018] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
INFO[0021] Cluster 'demo' created successfully!         
INFO[0021] You can now use it like this:                
kubectl cluster-info

❯ flux install
✚ generating manifests
✔ manifests build completed
► installing components in flux-system namespace
CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io created
CustomResourceDefinition/buckets.source.toolkit.fluxcd.io created
CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io created
CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io created
CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io created
CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io created
CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io created
CustomResourceDefinition/providers.notification.toolkit.fluxcd.io created
CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io created
Namespace/flux-system created
ServiceAccount/flux-system/helm-controller created
ServiceAccount/flux-system/kustomize-controller created
ServiceAccount/flux-system/notification-controller created
ServiceAccount/flux-system/source-controller created
ClusterRole/crd-controller-flux-system created
ClusterRoleBinding/cluster-reconciler-flux-system created
ClusterRoleBinding/crd-controller-flux-system created
Service/flux-system/notification-controller created
Service/flux-system/source-controller created
Service/flux-system/webhook-receiver created
Deployment/flux-system/helm-controller created
Deployment/flux-system/kustomize-controller created
Deployment/flux-system/notification-controller created
Deployment/flux-system/source-controller created
NetworkPolicy/flux-system/allow-egress created
NetworkPolicy/flux-system/allow-scraping created
NetworkPolicy/flux-system/allow-webhooks created
◎ verifying installation
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ install finished

❯ kgp -n flux-system
NAME                                      READY   STATUS    RESTARTS   AGE
notification-controller-d9cc9bf46-tjf4s   1/1     Running   0          45s
helm-controller-dfb4b5478-gxp5b           1/1     Running   0          45s
source-controller-84bfd77bf8-bjqgd        1/1     Running   0          45s
kustomize-controller-cd544c8f8-q857m      1/1     Running   0          45s

❯ cat<<EOF | kubectl apply -f -
pipe heredoc> apiVersion: <http://source.toolkit.fluxcd.io/v1beta1|source.toolkit.fluxcd.io/v1beta1>
pipe heredoc> kind: HelmRepository
pipe heredoc> metadata:
pipe heredoc>   name: vault
pipe heredoc> spec:
pipe heredoc>   interval: 5m
pipe heredoc>   url: <https://helm.releases.hashicorp.com>
pipe heredoc> EOF                                                     
<http://helmrepository.source.toolkit.fluxcd.io/vault|helmrepository.source.toolkit.fluxcd.io/vault> created

kg <http://helmrepositories.source.toolkit.fluxcd.io|helmrepositories.source.toolkit.fluxcd.io> vault
NAME    URL                                   AGE   READY   STATUS
vault   <https://helm.releases.hashicorp.com>   19s   False   failed to fetch Helm repository index: failed to cache index to temporary file: Get "<https://helm.releases.hashicorp.com/index.yaml>": dial tcp: lookup <http://helm.releases.hashicorp.com|helm.releases.hashicorp.com> on 10.43.0.10:53: server misbehaving

What about the CoreDNS pod? 🤔 And maybe you can exec into some pod and try to

nslookup <http://helm.releases.hashicorp.com|helm.releases.hashicorp.com>

?

sure sorry coredns-d76bd69b-ttdxv 1/1 Running 0 14m klo coredns-d76bd69b-ttdxv [ERROR] plugin/errors: 2 helm.releases.hashicorp.com. AAAA: read udp 10.42.0.5:34419->8.8.8.8:53: i/o timeout [ERROR] plugin/errors: 2 helm.releases.hashicorp.com. A: read udp 10.42.0.5:55438->8.8.8.8:53: i/o timeout [ERROR] plugin/errors: 2 helm.releases.hashicorp.com. A: read udp 10.42.0.5:42502->8.8.8.8:53: i/o timeout [ERROR] plugin/errors: 2 helm.releases.hashicorp.com. AAAA: read udp 10.42.0.5:42314->8.8.8.8:53: i/o timeout [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.serve

Are you behind a corporate firewall or something like this? Can you ping 8.8.8.8 locally?

yes I am sitting behind a coporate firewall. But 2 reasons why I dont think this is the problem is ping works to google: root@curlpod:/ ]$ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=115 time=21.385 ms When I am creating a Kind Cluster / minikube it works kcuc kind-vault Switched to context "kind-vault". ❯ kg helmrepositories.source.toolkit.fluxcd.io NAME URL AGE READY STATUS vault https://helm.releases.hashicorp.com 6m55s True stored artifact for revision 'fd98f29559cbb6bfa028e9ce533d6d5915aa3aa7c6a03095b95fcc2f14a53a30'

Can you try once to create the cluster with

K3D_FIX_DNS=1 k3d cluster create

and try that please?

❯ K3D_FIX_DNS=1 k3d cluster create INFO[0000] Prep: Network INFO[0000] Created network 'k3d-k3s-default' INFO[0000] Created image volume k3d-k3s-default-images INFO[0000] Starting new tools node... INFO[0000] Starting Node 'k3d-k3s-default-tools' INFO[0001] Creating node 'k3d-k3s-default-server-0' INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb' INFO[0001] Using the k3d-tools node to gather environment information INFO[0002] HostIP: using network gateway 172.24.0.1 address INFO[0002] Starting cluster 'k3s-default' INFO[0002] Starting servers... INFO[0002] Starting Node 'k3d-k3s-default-server-0' INFO[0010] All agents already running. INFO[0010] Starting helpers... INFO[0011] Starting Node 'k3d-k3s-default-serverlb' INFO[0018] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... INFO[0020] Cluster 'k3s-default' created successfully! INFO[0020] You can now use it like this: kubectl cluster-info ❯ kgpo No resources found in default namespace. ❯ kubectl run curlpod --image=radial/busyboxplus:curl --command -- /bin/sh -c "while true; do echo hi; sleep 10; done" pod/curlpod created ❯ kex curlpod -- sh [ root@curlpod:/ ]$ nslookup helmrepositories.source.toolkit.fluxcd.io Server: 10.43.0.10 Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local nslookup: can't resolve 'helmrepositories.source.toolkit.fluxcd.io' [ root@curlpod:/ ]$ I

I think that's the wrong nslookup there 😬

I found something maybe interesting: in the working kind cluster I got also for nslokup nslookup: can't resolve So i tried to install the stuff into the k3d with the DNS fix and Tata 😹 its working: ❯ kg helmrepositories.source.toolkit.fluxcd.io NAME URL AGE READY STATUS vault https://helm.releases.hashicorp.com 6s True stored artifact for revision 'fd98f29559cbb6bfa028e9ce533d6d5915aa3aa7c6a03095b95fcc2f14a53a30' Thanks a lot for your help 👍

I guess that's because you tried to resolve

<http://helmrepositories.source.toolkit.fluxcd.io|helmrepositories.source.toolkit.fluxcd.io>

which is the API of Flux, instead of

<https://helm.releases.hashicorp.com>

which is the repo URL. Cool that it works now, you're welcome 🙂