Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
s
sticky-art-85060
07/11/2022, 10:16 AMhi all ... i have a very basic question ... why all my pods are hang in ContainerCreating ... where can I check for the local issue?
$kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
local-path-provisioner-6c79684f77-lk6x7 0/1 ContainerCreating 0 110s
helm-install-traefik-pfml6 0/1 ContainerCreating 0 110s
helm-install-traefik-crd-8skws 0/1 ContainerCreating 0 110s
coredns-d76bd69b-nc2q7 0/1 ContainerCreating 0 110s
metrics-server-7cd5fcb6b7-mr99d 0/1 ContainerCreating 0 110s
m
miniature-advantage-69986
07/11/2022, 10:19 AMhave you looked at the events ( kubectl -n kube-system get events) or else kubectl -n kube-system describe pod helm-install-traefik-pfml6 ?
You just might have a problem pulling images. Do you use a proxy for accessing the internet?
s
sticky-art-85060
07/11/2022, 10:26 AM😮 yep ...
12m Warning FailedCreatePodSandBox pod/helm-install-traefik-pfml6 Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 172.20.65.95:53: read udp 172.20.65.95:55538->172.20.65.95:53: i/o timeout
11m Warning FailedCreatePodSandBox pod/coredns-d76bd69b-nc2q7 Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.6": Error response from daemon: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 172.20.65.95:53: read udp 172.20.65.95:45741->172.20.65.95:53: i/o timeout
is it somehow possible to configure our "company cr". I believe I'm not allowed to fetch images directly from the internet ... security 😉
m
miniature-advantage-69986
07/11/2022, 10:43 AMwell that depends on how your "company cr" is defined. Does docker/nerctl pull ... work? You might have to configure a proxy by setting "https_proxy" or "http_proxy" in RD
s
sticky-art-85060
07/11/2022, 11:52 AM@miniature-advantage-69986 I will try that with the proxy ... then maybe rd/docker is able to fetch some additional images from the internet. How can I do that? I'm under windows. But I guess finally I need a setup where only the company image-registry is used. Is there a howto or something? thanks!
m
miniature-advantage-69986
07/11/2022, 1:07 PM@sticky-art-85060 i haven't tried it my self but you could try this: Have a look at the documentation for k3s at https://rancher.com/docs/k3s/latest/en/installation/private-registry/ after changing the configuration you will probably have to restart Rancher Desktop
👍 1
s
sticky-art-85060
07/13/2022, 4:55 AMNoob question @miniature-advantage-69986 ... the docu you posted are explain something for rancher ... I'm runnning rancher-desktop and currently I don't plan to utilize rancher. is the setup regarding private registry also true for "only" rancher-desktop?
m
miniature-advantage-69986
07/13/2022, 6:20 AMThe documentation concerns k3s (a lightweight opensource kubernetes developed by Rancher) and has nothing to do with the product Rancher. Rancher Desktop is another opensource project developed by Rancher and it uses k3s. So the setup regarding private registries is true for k3s which is part of Rancher Desktop.
You wrote that you are using Windows, you can look inside of Rancher-Desktop with the this cmd "wsl -d rancher-desktop" and there you'll find the directory "/etc/rancher/k3s"
s
sticky-art-85060
07/13/2022, 8:49 AMI think my proxy-issue is this ticket here
https://github.com/rancher-sandbox/rancher-desktop/issues/2259
But until yet there is not really an easy answer or?
m
miniature-advantage-69986
07/13/2022, 12:52 PM@sticky-art-85060 can you access docker with curl in Windows "`curl.exe -vk https://registry-1.docker.io/v2/"` " and do you get the answer "401 Unauthorized"?
s
sticky-art-85060
07/13/2022, 1:12 PMIn my windows gitbash I got
156{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
m
miniature-advantage-69986
07/13/2022, 1:13 PMOK, and do have a proxy set in Windows?
s
sticky-art-85060
07/13/2022, 1:14 PMBut I can build docker images locally (I'm using maven jib plugin) ... so pulling from remote seems to work there.
m
miniature-advantage-69986
07/13/2022, 1:14 PMand does docker pull work in windows too?
s
sticky-art-85060
07/13/2022, 1:28 PMHmm .. I'm not really sure 😞
I'm behind a company proxy ... here I'm rel. sure 🙂
If I run "docker pull echo" ... with no proxy setting in my gitbash (windows)... I got
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 172.20.68.142:53: read udp 172.20.68.142:54694->172.20.68.142:53: i/o timeout
If I define HTTP_PROXY HTTPS_PROXY env variables ... then the docker pull simply hangs ... take forever
Maybe out-of-topic ... but if I run the maven jib plugin to build the docker image of my app .... rancher-desktop is runnng ... it works
[INFO] --- jib-maven-plugin:3.2.1:dockerBuild (default-cli) @ rest-service-sce ---
[INFO]
[INFO] Containerizing application to Docker daemon as cr.siemens.com/ebconf/spice-configuration-cluster/ccsce:2022.6.0-SNAPSHOT...
[WARNING] Base image 'eclipse-temurin:8-jre' does not use a specific image digest - build may not be reproducible
[INFO] The base image requires auth. Trying again for eclipse-temurin:8-jre...
[WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: registry-1.docker.io
[WARNING]
Got output:
credentials not found in native keychain
[WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: registry.hub.docker.com
[WARNING]
Got output:
credentials not found in native keychain
[WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: index.docker.io
[WARNING]
Got output:
credentials not found in native keychain
[WARNING] The credential helper (docker-credential-wincred) has nothing for server URL: docker.io
[WARNING]
Got output:
credentials not found in native keychain
[INFO] Using base image with digest: sha256:d6795e3c43f894332b3cee13dd8d5ddebe4bba382d41e454ba0451c1a80dc6c5
Another experiment ... I switched from dockerd to containerd in my rancher-desktop...
$winpty nerdctl run -it debian
docker.io/library/debian:latest: resolving |--------------------------------------|
elapsed: 9.9 s total: 0.0 B (0.0 B/s)
INFO[0010] trying next host error="failed to do request: Head \"https://registry-1.docker.io/v2/library/debian/manifests/latest\": dial tcp: lookup registry-1.docker.io on 172.20.68.142:53: read udp 172.20.68.142:39442->172.20.68.142:53: i/o timeout" host=registry-1.docker.io
ERRO[0010] active check failed error="context canceled"
FATA[0010] failed to resolve reference "docker.io/library/debian:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/debian/manifests/latest": dial tcp: lookup registry-1.docker.io on 172.20.68.142:53: read udp 172.20.68.142:39442->172.20.68.142:53: i/o timeout
m
miniature-advantage-69986
07/13/2022, 2:27 PMIt looks like your DNS is not being forwarded, it could be blocked by a local firewall in Windows.
s
sticky-art-85060
07/14/2022, 7:30 AMI checked /etc/resolv.conf in rancher-desktop machine (wsl -d rancher-desktop)
and found only one entry 172.20.xx.xxx ... I believe this is something from my company network,
I'm also forced to use zscaler vpn from homeoffice etc. .. maybe this is also an issue.
So inside rd ... a curl -vv http://www.heise.de failed because of no dns
Then I added 8.8.8.8 as another nameserver .. and then curl worked.
But docker pull debian etc. (from windows gitbash) still doesnt work ...
So I guess this was not a fix or something.
m
miniature-advantage-69986
07/14/2022, 11:42 AMI think that @fast-garage-66093 will be able to tell us how exactly dns requests are forwarded in RD. Where you abel to find out which proxy you must configure?
f
fast-garage-66093
07/14/2022, 4:33 PMIs this on Windows? In that case you should enable the experimental host resolver first: Epic: Use Lima host resolver on Windows · Issue #1899 · rancher-sandbox/rancher-desktop
That should solve the DNS issues with the VPN because it will lookup names on the host and not inside the VM
I don't remember how you can configure the proxies manually on Windows (for RD), and it may depend on if you are using containerd or moby
w
wide-mechanic-33041
07/18/2022, 11:57 AMnot much progress for windows. lots of issues like https://github.com/rancher-sandbox/rancher-desktop/issues/2259. Can’t figure out how to exclude the internal networking so k8s starts throwing errors and timeout on start.