Hi everyone, I use rke to create the K8s cluster, and I use network provider calico to archive pod communication within the cluster but I have an issue with it: The calico-node pod in some nodes restarted frequently My cluster.yaml

network:
  plugin: calico
addons: |-
  ---
  apiVersion: v1
  data:
    calico_backend: bird
    cni_network_config: |-
      {
        "name": "k8s-pod-network",
        "cniVersion": "0.3.1",
        "plugins": [
          {
            "type": "calico",
            "log_level": "WARNING",
            "log_file_path": "/var/log/calico/cni/cni.log",
            "datastore_type": "kubernetes",
            "nodename": "__KUBERNETES_NODE_NAME__",
            "mtu": __CNI_MTU__,
            "ipam": {
                "type": "calico-ipam"
            },
            "policy": {
                "type": "k8s"
            },
            "kubernetes": {
                "kubeconfig": "__KUBECONFIG_FILEPATH__"
            }
          },
          {
            "type": "portmap",
            "snat": true,
            "capabilities": {"portMappings": true}
          },
          {
            "type": "bandwidth",
            "capabilities": {"bandwidth": true}
          }
        ]
      }
    typha_service_name: none
    veth_mtu: "1430"
  kind: ConfigMap
  metadata:
    name: calico-config
    namespace: kube-system

Nothing obvious to me in there. What does

describe

for that pod say?

My restarted pod inspection:

Init Containers:
  upgrade-ipam:
    Container ID:  <containerd://7d8572faf69a3a684fb99046471d632b8717f5c0bbe3ee5b48ba829f68ffc0f>9
    Image:         rancher/calico-cni:v3.17.2
    Image ID:      <http://docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb|docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb>
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 21 Dec 2022 09:35:41 +0700
      Finished:     Wed, 21 Dec 2022 09:35:44 +0700
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
  install-cni:
    Container ID:  <containerd://3906b7c0354e2051b8d181b6d27f5d657b11227c10282c88fb31f36200762c3>4
    Image:         rancher/calico-cni:v3.17.2
    Image ID:      <http://docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb|docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb>
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 21 Dec 2022 09:36:15 +0700
      Finished:     Wed, 21 Dec 2022 09:36:36 +0700
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
  flexvol-driver:
    Container ID:   <containerd://247a099ee96e806b6472f5a6e48dfba04d7683ffdfecdbc5994cf44dc473354>4
    Image:          rancher/calico-pod2daemon-flexvol:v3.17.2
    Image ID:       <http://docker.io/rancher/calico-pod2daemon-flexvol@sha256:adbe9ea3e36587828cf0fd2c008029052ad893abf923e2200ac0746b25a77248|docker.io/rancher/calico-pod2daemon-flexvol@sha256:adbe9ea3e36587828cf0fd2c008029052ad893abf923e2200ac0746b25a77248>
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 21 Dec 2022 09:37:25 +0700
      Finished:     Wed, 21 Dec 2022 09:37:26 +0700
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
Containers:
  calico-node:
    Container ID:   <containerd://908b7ca9fb179cba7a5e20475e8ca88e47323e175430d7c1b96eec7a20d0dcc>1
    Image:          rancher/calico-node:v3.17.2
    Image ID:       <http://docker.io/rancher/calico-node@sha256:6ba192911e28d052da5d830ff80521dfcb7444c886028795d94133914e187c6a|docker.io/rancher/calico-node@sha256:6ba192911e28d052da5d830ff80521dfcb7444c886028795d94133914e187c6a>
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 21 Dec 2022 17:19:59 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 21 Dec 2022 17:18:09 +0700
      Finished:     Wed, 21 Dec 2022 17:18:12 +0700
    Ready:          True
    Restart Count:  34
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               172.22.0.0/16
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGFILEPATH:                  none
      FELIX_LOGSEVERITYSYS:
      FELIX_LOGSEVERITYSCREEN:            Warning
      FELIX_HEALTHENABLED:                true
      FELIX_IPTABLESBACKEND:              auto
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/ from sysfs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sysfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-fd22m:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-fd22m
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <http://kubernetes.io/os=linux|kubernetes.io/os=linux>
Tolerations:     :NoSchedule op=Exists
                 :NoExecute op=Exists
                 CriticalAddonsOnly op=Exists
                 <http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule> op=Exists
                 <http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule> op=Exists
                 <http://node.kubernetes.io/network-unavailable:NoSchedule|node.kubernetes.io/network-unavailable:NoSchedule> op=Exists
                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists
                 <http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule> op=Exists
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists
                 <http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule> op=Exists
Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Warning  Unhealthy  36m (x312 over 8h)    kubelet  (combined from similar events): Readiness probe failed:
  Warning  Unhealthy  6m11s (x382 over 8h)  kubelet  Readiness probe failed:
  Warning  Unhealthy  32s (x540 over 8h)    kubelet  Liveness probe failed:

Liveness probe failed:

So looks like it failed a Liveness check and was killed by kubelet. Are there any logs about

liveness

?

this is all information when I describe the pod But why kubelet cannot do Readiness and Liveness

But why kubelet cannot do Readiness and Liveness

Yes, that's what I'm wondering too. Are there any logs from the pod with

liveness

or

readiness

in them?

Nope, I didn't have any configuration related to resources, only CPU requests was setup (0.25) but I think it is default

I guess you're looking for "Liveness probe" and probably the name of the pod

this is the log including

calico

in the last 1000 lines of kubelet log sorry, I cannot send logs in text because some characters are too long.

Deadline Exceeded

is the error. i.e. it tried to run the Liveness command in the calico-node pod, but it failed to return within the deadline. Is your system heavily loaded (any free CPU)? How large is it? How many pods per node? How many nodes?

@great-jewelry-76121 tks u, I will check it and reply u later

137 pods on this node? Or on the cluster?

just on this node

And how much CPU is in use?

Current Load average is 18.99. Does this box have more than 19 CPUs?

I think I only have 8 CPUs as the screenshot

https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes:

Some interpretations:

• If the averages are 0.0, then your system is idle.

• If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.

• If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.

• If they are higher than your CPU count, then you might have a performance problem (it depends).

oh, tks for useful knowledge

Yes. If calico-node doesn't get CPU in time to respond to Liveness checks, it will get restarted. Options: • put CPU limits on pods on this node. • put fewer pods on this node - which may mean adding more nodes to the cluster Be a little careful about putting CPU requests and limits on calico-node. calico-node CPU usage is very bursty - it does a lot of work when it is starting up (loading the state of the node) or when pods are created or deleted, very little at other times. If you set the limit too low, it may not start up fast enough, and kubelet will then kill it, causing it to start up again - basically a self-induced crashloop!

oh, nice, I think I've already met the right person 🙂 I will try to find out the solution following your suggestion Once again, thank u for all insight u gave me I will try to reduce the load average and report to u the result

No worries, glad to be able to help

hi @great-jewelry-76121, now, I cannot reduce the load average so I decided to increase the timeoutSeconds within the calico-node daemonset (from 1 second to 10 seconds) the restarting phenomenon stopped but the liveness probe + readiness probe failed events still happen. Does it mean 10 seconds timeout wasn't enough for probe?

Looking at https://docs.projectcalico.org/manifests/calico.yaml, I see the liveness/readiness sections say:

livenessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-live
              - -bird-live
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-ready
              - -bird-ready
            periodSeconds: 10
            timeoutSeconds: 10

i.e. it shouldn't have been set to 1 second anyway.

Does it mean 10 seconds timeout wasn't enough for probe?

Yes, assuming that the restart was due to Liveness.

I haven't met any critical problems, but my API gateway daemonset also was restarted and the pod is in the same with restarted calico-node

As I know, calico-node uses iptables to route so what calico's components attend to the datapath?

The calico-node pod: • installs the CNI plugin, which networks pods when they are created, and cleans up when they are deleted • updates iptables rules on the node to implement network policy

@great-jewelry-76121 thank u I wanna learn more about Calico, where should I start? And does Calico project have QA forum like this?

https://slack.projectcalico.org/ for the Calico slack https://academy.tigera.io/course/certified-calico-operator-level-1/ for the free Calico Certified Operator L1 course.

I have some critical problems with my Calico deployment, I hope it will help to find out the solution 😄