hi all i have a very basic question why all my pods are hang rancher-users #rancher-desktop

Title

# rancher-desktop

sticky-art-85060

07/11/2022, 10:16 AM

hi all ... i have a very basic question ... why all my pods are hang in ContainerCreating ... where can I check for the local issue? $kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE local-path-provisioner-6c79684f77-lk6x7 0/1 ContainerCreating 0 110s helm-install-traefik-pfml6 0/1 ContainerCreating 0 110s helm-install-traefik-crd-8skws 0/1 ContainerCreating 0 110s coredns-d76bd69b-nc2q7 0/1 ContainerCreating 0 110s metrics-server-7cd5fcb6b7-mr99d 0/1 ContainerCreating 0 110s

miniature-advantage-69986

07/11/2022, 10:19 AM

have you looked at the events ( kubectl -n kube-system get events) or else kubectl -n kube-system describe pod helm-install-traefik-pfml6 ?

You just might have a problem pulling images. Do you use a proxy for accessing the internet?

sticky-art-85060

07/11/2022, 10:26 AM

😮 yep ... 12m Warning FailedCreatePodSandBox pod/helm-install-traefik-pfml6 Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 172.20.65.9553 read udp 172.20.65.9555538 >172.20.65.9553: i/o timeout 11m Warning FailedCreatePodSandBox pod/coredns-d76bd69b-nc2q7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 172.20.65.9553 read udp 172.20.65.9545741 >172.20.65.9553: i/o timeout

is it somehow possible to configure our "company cr". I believe I'm not allowed to fetch images directly from the internet ... security 😉

miniature-advantage-69986

07/11/2022, 10:43 AM

well that depends on how your "company cr" is defined. Does docker/nerctl pull ... work? You might have to configure a proxy by setting "https_proxy" or "http_proxy" in RD

sticky-art-85060

07/11/2022, 11:52 AM

@miniature-advantage-69986 I will try that with the proxy ... then maybe rd/docker is able to fetch some additional images from the internet. How can I do that? I'm under windows. But I guess finally I need a setup where only the company image-registry is used. Is there a howto or something? thanks!

miniature-advantage-69986

07/11/2022, 1:07 PM

@sticky-art-85060 i haven't tried it my self but you could try this: Have a look at the documentation for k3s at https://rancher.com/docs/k3s/latest/en/installation/private-registry/ after changing the configuration you will probably have to restart Rancher Desktop

👍 1

sticky-art-85060

07/13/2022, 4:55 AM

Noob question @miniature-advantage-69986 ... the docu you posted are explain something for rancher ... I'm runnning rancher-desktop and currently I don't plan to utilize rancher. is the setup regarding private registry also true for "only" rancher-desktop?

miniature-advantage-69986

07/13/2022, 6:20 AM

The documentation concerns k3s (a lightweight opensource kubernetes developed by Rancher) and has nothing to do with the product Rancher. Rancher Desktop is another opensource project developed by Rancher and it uses k3s. So the setup regarding private registries is true for k3s which is part of Rancher Desktop. You wrote that you are using Windows, you can look inside of Rancher-Desktop with the this cmd "wsl -d rancher-desktop" and there you'll find the directory "/etc/rancher/k3s"

sticky-art-85060

07/13/2022, 8:49 AM

I think my proxy-issue is this ticket here https://github.com/rancher-sandbox/rancher-desktop/issues/2259 But until yet there is not really an easy answer or?

miniature-advantage-69986

07/13/2022, 12:52 PM

@sticky-art-85060 can you access docker with curl in Windows "`curl.exe -vk https://registry-1.docker.io/v2/"` " and do you get the answer "401 Unauthorized"?

sticky-art-85060

07/13/2022, 1:12 PM

In my windows gitbash I got 156{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

miniature-advantage-69986

07/13/2022, 1:13 PM

OK, and do have a proxy set in Windows?

sticky-art-85060

07/13/2022, 1:14 PM

But I can build docker images locally (I'm using maven jib plugin) ... so pulling from remote seems to work there.

miniature-advantage-69986

07/13/2022, 1:14 PM

and does docker pull work in windows too?

sticky-art-85060

07/13/2022, 1:28 PM

Hmm .. I'm not really sure 😞 I'm behind a company proxy ... here I'm rel. sure 🙂 If I run "docker pull echo" ... with no proxy setting in my gitbash (windows)... I got Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 172.20.68.14253 read udp 172.20.68.14254694 >172.20.68.14253: i/o timeout If I define HTTP_PROXY HTTPS_PROXY env variables ... then the docker pull simply hangs ... take forever

Maybe out-of-topic ... but if I run the maven jib plugin to build the docker image of my app .... rancher-desktop is runnng ... it works [INFO] --- jib-maven-plugin3.2.1dockerBuild (default-cli) @ rest-service-sce --- [INFO] [INFO] Containerizing application to Docker daemon as cr.siemens.com/ebconf/spice-configuration-cluster/ccsce:2022.6.0-SNAPSHOT... [WARNING] Base image 'eclipse-temurin:8-jre' does not use a specific image digest - build may not be reproducible [INFO] The base image requires auth. Trying again for eclipse-temurin:8-jre... [WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: registry-1.docker.io [WARNING] Got output: credentials not found in native keychain [WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: registry.hub.docker.com [WARNING] Got output: credentials not found in native keychain [WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: index.docker.io [WARNING] Got output: credentials not found in native keychain [WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: docker.io [WARNING] Got output: credentials not found in native keychain [INFO] Using base image with digest: sha256:d6795e3c43f894332b3cee13dd8d5ddebe4bba382d41e454ba0451c1a80dc6c5

Another experiment ... I switched from dockerd to containerd in my rancher-desktop... $winpty nerdctl run -it debian docker.io/library/debian:latest: resolving |--------------------------------------| elapsed: 9.9 s total: 0.0 B (0.0 B/s) INFO[0010] trying next host error="failed to do request: Head \"https://registry-1.docker.io/v2/library/debian/manifests/latest\": dial tcp: lookup registry-1.docker.io on 172.20.68.14253 read udp 172.20.68.14239442 >172.20.68.14253: i/o timeout" host=registry-1.docker.io ERRO[0010] active check failed error="context canceled" FATA[0010] failed to resolve reference "docker.io/library/debian:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/debian/manifests/latest": dial tcp: lookup registry-1.docker.io on 172.20.68.14253 read udp 172.20.68.14239442 >172.20.68.14253: i/o timeout

miniature-advantage-69986

07/13/2022, 2:27 PM

It looks like your DNS is not being forwarded, it could be blocked by a local firewall in Windows.

sticky-art-85060

07/14/2022, 7:30 AM

I checked /etc/resolv.conf in rancher-desktop machine (wsl -d rancher-desktop) and found only one entry 172.20.xx.xxx ... I believe this is something from my company network, I'm also forced to use zscaler vpn from homeoffice etc. .. maybe this is also an issue. So inside rd ... a curl -vv http://www.heise.de failed because of no dns Then I added 8.8.8.8 as another nameserver .. and then curl worked. But docker pull debian etc. (from windows gitbash) still doesnt work ... So I guess this was not a fix or something.

miniature-advantage-69986

07/14/2022, 11:42 AM

I think that @fast-garage-66093 will be able to tell us how exactly dns requests are forwarded in RD. Where you abel to find out which proxy you must configure?

fast-garage-66093

07/14/2022, 4:33 PM

Is this on Windows? In that case you should enable the experimental host resolver first: Epic: Use Lima host resolver on Windows · Issue #1899 · rancher-sandbox/rancher-desktop

That should solve the DNS issues with the VPN because it will lookup names on the host and not inside the VM

I don't remember how you can configure the proxies manually on Windows (for RD), and it may depend on if you are using containerd or moby

wide-mechanic-33041

07/18/2022, 11:57 AM

not much progress for windows. lots of issues like https://github.com/rancher-sandbox/rancher-desktop/issues/2259. Can’t figure out how to exclude the internal networking so k8s starts throwing errors and timeout on start.

9 Views