https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
c
Do you have more than one interface with a default route, or no default route at all?
m
The issue apparently went away after I passed 
--node-ip
, 
--node-external-ip
, and 
--flannel-iface
with the IP address assigned by the cloud env. However, now getting following errors:
Copy code
time="2022-06-30T00:36:18+09:00" level=info msg="Cluster-Http-Server 2022/06/30 00:36:18 http: TLS handshake error from 127.0.0.1:50128: remote error: tls: bad certificate"
time="2022-06-30T00:36:18+09:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: unable to find interface: route ip+net: no such network interface"
c
Same question
You may need to add a dummy default route. Kubernetes can be run airgapped but it still expects to have a default route for a bunch of the interface selection logic.
m
ip route
shows one default route: 
default nhid 12 via inet6 f0::...... proto bgp src <internal ipv4 address> metric 20
changed --flannel-iface to interface name instead of ip. Seems to work now.
Now getting following errors:
Copy code
E0630 01:03:27.777938  182940 resource_quota_controller.go:413] unable to retrieve the complete list of server APIs: <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request

W0630 01:03:28.050326  182940 garbagecollector.go:707] failed to discover some groups: map[<http://metrics.k8s.io/v1beta1:the|metrics.k8s.io/v1beta1:the> server is currently unable to handle the request]

E0630 01:03:33.398952  182940 pod_workers.go:951] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"lb-tcp-80\" with CrashLoopBackOff: \"back-off 20s restarting failed container=lb-tcp-80 pod=svclb-traefik-dc729b50-vrfdm_kube-system(0e80293c-9e5c-4a08-9126-1e4515fed5fd)\", failed to \"StartContainer\" for \"lb-tcp-443\" with CrashLoopBackOff: \"back-off 20s restarting failed container=lb-tcp-443 pod=svclb-traefik-dc729b50-vrfdm_kube-system(0e80293c-9e5c-4a08-9126-1e4515fed5fd)\"]" pod="kube-system/svclb-traefik-dc729b50-vrfdm" podUID=0e80293c-9e5c-4a08-9126-1e4515fed5fd

E0630 01:03:40.610544  182940 available_controller.go:524] <http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io> failed with: failing or missing response from <https://10.42.0.13:4443/apis/metrics.k8s.io/v1beta1>: Get "<https://10.42.0.13:4443/apis/metrics.k8s.io/v1beta1>": proxy error from 127.0.0.1:6443 while dialing 10.42.0.13:4443, code 500: 500 Internal Server Error
c
If you describe the crashing pods, and look at the logs, what does it say?
m
@creamy-pencil-82913 `kubectl logs --namespace=kube-system coredns-d76bd69b-qnvxd`:
Copy code
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
.:53
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d
CoreDNS-1.9.1
linux/amd64, go1.17.8, 4b597f8
[ERROR] plugin/errors: 2 3162228087811675511.9106375430785523333. HINFO: read udp 10.42.0.24:36527->8.8.8.8:53: i/o timeout
`kubectl logs --namespace=kube-system traefik-df4ff85d6-4jd6w`:
Copy code
time="2022-06-29T23:40:50Z" level=info msg="Configuration loaded from flags."
time="2022-06-29T23:41:02Z" level=error msg="accept tcp [::]:9100: use of closed network connection" entryPointName=metrics
time="2022-06-29T23:41:02Z" level=error msg="accept tcp [::]:9000: use of closed network connection" entryPointName=traefik
time="2022-06-29T23:41:02Z" level=error msg="accept tcp [::]:8443: use of closed network connection" entryPointName=websecure
time="2022-06-29T23:41:02Z" level=error msg="accept tcp [::]:8000: use of closed network connection" entryPointName=web
time="2022-06-29T23:41:02Z" level=error msg="close tcp [::]:9000: use of closed network connection" entryPointName=traefik
time="2022-06-29T23:41:02Z" level=error msg="close tcp [::]:8443: use of closed network connection" entryPointName=websecure
time="2022-06-29T23:41:02Z" level=error msg="close tcp [::]:9100: use of closed network connection" entryPointName=metrics
time="2022-06-29T23:41:02Z" level=error msg="close tcp [::]:8000: use of closed network connection" entryPointName=web
c
looks like it can’t talk to DNS servers…
for coredns at leastm not sure why traefik is crashing
m
the environment has limitations regarding internet access, so in case it tries to access internet, that might be cause
How can I override this? I don’t want k3s to connect to google dns, nor any other external DNS for that matter
how do I configure coredns/k3s to use specific DNS server(s)?
hmm, this is getting tricky.. I need to also pull all the images from internal Harbor registry
by specifying resolv.conf, it seems all pods are now up and running
20 Views
Open in Slack
PreviousNext
Powered by