https://rancher.com/ logo
Join Slack
Powered by
No. Flannel does not run as a pod. flannel and kub...
# general
c
No. Flannel does not run as a pod. flannel and kube-router network policy controller run in the main k3s process, same as etcd, kubelet, apiserver, and all the other core components.
b
Yes, I can see in the logs, that flannel is started. In the logs of K3S I can see that many services get an context deadline exceeded or timeout. What should I check for to get it working?
c
I don’t know, that would really depend on whats failing that is causing things to time out.
b
Something network related I suppose. No service can reach any IP in the cluster neither through an IP or a DNS name. Services in a single pod can communicate just fine. Also external IPs and the service behind them can be reached. Examples: Aug 22 195701 k8s-vm1 k3s[106276]: E0822 195701.964027 106276 remote_available_controller.go:448] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.1.41:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.42.1.41:10250/apis/metrics.k8s.io/v1beta1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" │ django.db.utils.OperationalError: connection timeout expired
c
sounds like your CNI is broken. Did you disable your firewall? is the vxlan kernel module loaded?
If this is on a vmware vm, have you disabled checksum offload?
b
This is on a proxmox vm. Firewall is disabled. lsmod | grep vxlan vxlan 155648 0 ip6_udp_tunnel 16384 1 vxlan udp_tunnel 36864 1 vxlan
c
is the metrics-server pod running? Do you have anything else managing iptables rules on this node, such as docker or a system-level installation of containerd?
b
Yes kube-system metrics-server-6f4c6675d5-gk2np 1/1 Running 0 31m And nothing else is installed or managing iptables. It is just the VM and the command that installed k3s
c
Did you see that error once, or does it continue to log that?
It is normal for there to be some errors during startup. Things aren’t run immediately, it takes time for them to start up and settle down
is there an actual problem that you are trying to diagnose?
b
I find about 100 lines of the same error.
c
does it go away eventually, once metrics-server is running and ready?
is there something that is not working, or are you just chasing logs
b
is there an actual problem that you are trying to diagnose?
Yes, I am trying to deploy paperless with postgres. Paperless crashes because it times out connecting to postgres. But I also tried with MariaDB Operator where a webhook could not be reached due to some type of networking issue. It does not work after all pods are started and not after hours.
does it go away eventually, once metrics-server is running and ready?
no
c
Does the pod or service IP range overlap with something on your network?
b
No, I only have a 192.168 network. The pods/services all have 10.x ips
c
If you have only a single node, there is not really much to check. Everything is local, there is not even anything going between nodes using vxlan or any other overlay network.
You are SURE you do not have anything else mucking about with iptables on this node?
no containerd, no docker, no ufw, no firewalld, no av/antimalware?
b
I have a second node in there aswell. NAME STATUS ROLES AGE VERSION k3s-vm2 Ready <none> 5d21h v1.33.3+k3s1 k8s-vm1 Ready control-plane,master 5d21h v1.33.3+k3s1 With IP 192.168.2.202/203.
You are SURE you do not have anything else mucking about with iptables on this node?
It is a clean installation of Rocky Linux 10. On this installation I disabled firewall as seen in the documentation and ran the quickstart script
I mean I can try again kicking the whole VM and starting over but this would be my last resort
c
do these VMs have multiple interfaces/IPs?
does the node IP if you do 
kubectl get node -o wide
show the correct IP that they can reach each other at?
b
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 0000000000:00 brd 0000000000:00 2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether bc241194b9:30 brd ffffffffff:ff altname enp0s18 altname enxbc241194b930 3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether 5e410299ec:70 brd ffffffffff:ff 4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether f6a0a87a20:a1 brd ffffffffff:ff 25: veth12ae3bb6@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default qlen 1000 link/ether e6d62de121:c6 brd ffffffffff:ff link-netns cni-4346ee98-0b9c-9347-9f29-c985cb0bf1fa 26: vethe28f562e@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default qlen 1000 link/ether 2a50c9a7bb:c8 brd ffffffffff:ff link-netns cni-bf67e007-3bd5-2fae-92b5-9e9b8d008bc9 27: veth4e169a6f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default qlen 1000 link/ether 665937eb5b:2e brd ffffffffff:ff link-netns cni-b205ebcc-2e0a-ff53-69ce-c3937dfeed8b 28: veth2cbe37dc@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default qlen 1000 link/ether 4a💿2ac128:b4 brd ffffffffff:ff link-netns cni-073c5509-e46c-df64-ed83-8b3aaa3efe59 This is from the vm1.
kubectl get node -o wide
It shows the correct IP and can be pinged
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k3s-vm2 Ready <none> 5d21h v1.33.3+k3s1 192.168.2.203 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.25.1.el10_0.x86_64 containerd://2.0.5-k3s2 k8s-vm1 Ready control-plane,master 5d22h v1.33.3+k3s1 192.168.2.202 <none> Rocky Linux 10.0 (Red Quartz) 6.12.0-55.25.1.el10_0.x86_64 containerd://2.0.5-k3s2
m
can it be something related to selinux ?
Open in Slack
PreviousNext
Powered by