many-animal-11139
12/21/2022, 9:47 AMnetwork:
plugin: calico
addons: |-
---
apiVersion: v1
data:
calico_backend: bird
cni_network_config: |-
{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"log_level": "WARNING",
"log_file_path": "/var/log/calico/cni/cni.log",
"datastore_type": "kubernetes",
"nodename": "__KUBERNETES_NODE_NAME__",
"mtu": __CNI_MTU__,
"ipam": {
"type": "calico-ipam"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "__KUBECONFIG_FILEPATH__"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
},
{
"type": "bandwidth",
"capabilities": {"bandwidth": true}
}
]
}
typha_service_name: none
veth_mtu: "1430"
kind: ConfigMap
metadata:
name: calico-config
namespace: kube-system
2022-12-21 15:20:10
2022-12-21 08:20:10.389 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:20:53
bird: KIF: Received address message for unknown interface 101074
2022-12-21 15:21:10
2022-12-21 08:21:10.395 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:22:07
bird: KIF: Received address message for unknown interface 101077
2022-12-21 15:22:10
2022-12-21 08:22:10.409 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:22:39
bird: KIF: Received address message for unknown interface 101078
2022-12-21 15:22:43
bird: KIF: Received address message for unknown interface 101083
2022-12-21 15:23:10
2022-12-21 08:23:10.414 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:23:23
bird: KIF: Received address message for unknown interface 101097
2022-12-21 15:24:10
2022-12-21 08:24:10.429 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:24:41
bird: KIF: Received address message for unknown interface 101068
2022-12-21 15:25:10
2022-12-21 08:25:10.435 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:25:45
bird: KIF: Received address message for unknown interface 101087
2022-12-21 15:26:10
2022-12-21 08:26:10.449 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:26:19
bird: KIF: Received address message for unknown interface 101102
2022-12-21 15:26:49
bird: KIF: Received address message for unknown interface 101101
2022-12-21 15:27:10
2022-12-21 08:27:10.457 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:27:17
bird: KIF: Received address message for unknown interface 101110
2022-12-21 15:28:10
2022-12-21 08:28:10.469 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:28:49
bird: KIF: Received address message for unknown interface 101115
2022-12-21 15:28:51
bird: KIF: Received address message for unknown interface 101114
2022-12-21 15:29:10
2022-12-21 08:29:10.475 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:30:05
bird: KRT: Received route 172.22.159.127/32 with unknown ifindex 101122
2022-12-21 15:30:10
2022-12-21 08:30:10.494 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:30:27
bird: KIF: Received address message for unknown interface 101108
2022-12-21 15:30:31
bird: KIF: Received address message for unknown interface 101107
2022-12-21 15:30:39
bird: KIF: Received address message for unknown interface 101117
2022-12-21 15:31:00
2022-12-21 08:31:00.322 [INFO][55] tunnel-ip-allocator/watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/ippools"
2022-12-21 15:31:10
2022-12-21 08:31:10.501 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:32:07
2022-12-21 08:32:07.049 [INFO][58] confd/watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/bgppeers"
2022-12-21 15:32:10
2022-12-21 08:32:10.512 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:32:29
bird: KIF: Received address message for unknown interface 101131
2022-12-21 15:33:10
2022-12-21 08:33:10.518 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:34:08
2022-12-21 08:34:08.003 [INFO][58] confd/watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/ippools"
2022-12-21 15:34:10
2022-12-21 08:34:10.541 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:34:35
bird: KIF: Received address message for unknown interface 101141
2022-12-21 15:35:10
2022-12-21 08:35:10.546 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:36:10
2022-12-21 08:36:10.557 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:36:23
bird: KIF: Received address message for unknown interface 101142
2022-12-21 15:36:45
bird: KIF: Received address message for unknown interface 101148
2022-12-21 15:36:49
bird: KIF: Received address message for unknown interface 101155
2022-12-21 15:37:10
2022-12-21 08:37:10.564 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:37:12
2022-12-21 08:37:12.524 [INFO][58] confd/watchercache.go 96: Watch channel closed by remote - recreate watcher ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2022-12-21 15:38:10
2022-12-21 08:38:10.571 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:39:10
2022-12-21 08:39:10.581 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:40:10
2022-12-21 08:40:10.591 [INFO][56] monitor-addresses/startup.go 759: Using autodetected IPv4 address on interface eth0: 10.10.0.44/24
2022-12-21 15:40:43
bird: KIF: Received address message for unknown interface 101159
great-jewelry-76121
12/21/2022, 10:36 AMdescribe
for that pod say?many-animal-11139
12/21/2022, 10:58 AMInit Containers:
upgrade-ipam:
Container ID: <containerd://7d8572faf69a3a684fb99046471d632b8717f5c0bbe3ee5b48ba829f68ffc0f>9
Image: rancher/calico-cni:v3.17.2
Image ID: <http://docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb|docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb>
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 21 Dec 2022 09:35:41 +0700
Finished: Wed, 21 Dec 2022 09:35:44 +0700
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
install-cni:
Container ID: <containerd://3906b7c0354e2051b8d181b6d27f5d657b11227c10282c88fb31f36200762c3>4
Image: rancher/calico-cni:v3.17.2
Image ID: <http://docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb|docker.io/rancher/calico-cni@sha256:903ab84bf707dda646cbe76b58e76953fddd6eac11ce33d06841e0781dc5a2bb>
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 21 Dec 2022 09:36:15 +0700
Finished: Wed, 21 Dec 2022 09:36:36 +0700
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
flexvol-driver:
Container ID: <containerd://247a099ee96e806b6472f5a6e48dfba04d7683ffdfecdbc5994cf44dc473354>4
Image: rancher/calico-pod2daemon-flexvol:v3.17.2
Image ID: <http://docker.io/rancher/calico-pod2daemon-flexvol@sha256:adbe9ea3e36587828cf0fd2c008029052ad893abf923e2200ac0746b25a77248|docker.io/rancher/calico-pod2daemon-flexvol@sha256:adbe9ea3e36587828cf0fd2c008029052ad893abf923e2200ac0746b25a77248>
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 21 Dec 2022 09:37:25 +0700
Finished: Wed, 21 Dec 2022 09:37:26 +0700
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
Containers:
calico-node:
Container ID: <containerd://908b7ca9fb179cba7a5e20475e8ca88e47323e175430d7c1b96eec7a20d0dcc>1
Image: rancher/calico-node:v3.17.2
Image ID: <http://docker.io/rancher/calico-node@sha256:6ba192911e28d052da5d830ff80521dfcb7444c886028795d94133914e187c6a|docker.io/rancher/calico-node@sha256:6ba192911e28d052da5d830ff80521dfcb7444c886028795d94133914e187c6a>
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 21 Dec 2022 17:19:59 +0700
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 21 Dec 2022 17:18:09 +0700
Finished: Wed, 21 Dec 2022 17:18:12 +0700
Ready: True
Restart Count: 34
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 172.22.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_LOGFILEPATH: none
FELIX_LOGSEVERITYSYS:
FELIX_LOGSEVERITYSCREEN: Warning
FELIX_HEALTHENABLED: true
FELIX_IPTABLESBACKEND: auto
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-fd22m (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
calico-node-token-fd22m:
Type: Secret (a volume populated by a Secret)
SecretName: calico-node-token-fd22m
Optional: false
QoS Class: Burstable
Node-Selectors: <http://kubernetes.io/os=linux|kubernetes.io/os=linux>
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
<http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/network-unavailable:NoSchedule|node.kubernetes.io/network-unavailable:NoSchedule> op=Exists
<http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists
<http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists
<http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule> op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 36m (x312 over 8h) kubelet (combined from similar events): Readiness probe failed:
Warning Unhealthy 6m11s (x382 over 8h) kubelet Readiness probe failed:
Warning Unhealthy 32s (x540 over 8h) kubelet Liveness probe failed:
great-jewelry-76121
12/21/2022, 11:00 AMLiveness probe failed:
So looks like it failed a Liveness check and was killed by kubelet.
Are there any logs about liveness
?many-animal-11139
12/21/2022, 11:01 AMgreat-jewelry-76121
12/21/2022, 11:02 AMBut why kubelet cannot do Readiness and LivenessYes, that's what I'm wondering too. Are there any logs from the pod with
liveness
or readiness
in them?many-animal-11139
12/21/2022, 11:05 AMgreat-jewelry-76121
12/21/2022, 11:16 AMmany-animal-11139
12/21/2022, 11:27 AMcalico
in the last 1000 lines of kubelet log
sorry, I cannot send logs in text because some characters are too long.great-jewelry-76121
12/21/2022, 11:29 AMDeadline Exceeded
is the error. i.e. it tried to run the Liveness command in the calico-node pod, but it failed to return within the deadline.
Is your system heavily loaded (any free CPU)? How large is it? How many pods per node? How many nodes?many-animal-11139
12/21/2022, 12:10 PMgreat-jewelry-76121
12/21/2022, 1:35 PMMay disk pressure impact?I don't think disk pressure should affect calico-node
many-animal-11139
12/21/2022, 1:36 PMgreat-jewelry-76121
12/21/2022, 1:37 PMtop
on the node itself?)many-animal-11139
12/21/2022, 1:41 PMgreat-jewelry-76121
12/21/2022, 1:42 PMmany-animal-11139
12/21/2022, 1:48 PMgreat-jewelry-76121
12/21/2022, 1:50 PMLinux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. Most tools show three averages, for 1, 5, and 15 minutes:
Some interpretations:
• If the averages are 0.0, then your system is idle.
• If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
• If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
• If they are higher than your CPU count, then you might have a performance problem (it depends).
many-animal-11139
12/21/2022, 1:52 PMgreat-jewelry-76121
12/21/2022, 1:59 PMmany-animal-11139
12/21/2022, 2:05 PMgreat-jewelry-76121
12/21/2022, 2:05 PMmany-animal-11139
12/23/2022, 2:45 AMgreat-jewelry-76121
12/23/2022, 9:44 AMlivenessProbe:
exec:
command:
- /bin/calico-node
- -felix-live
- -bird-live
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
timeoutSeconds: 10
readinessProbe:
exec:
command:
- /bin/calico-node
- -felix-ready
- -bird-ready
periodSeconds: 10
timeoutSeconds: 10
i.e. it shouldn't have been set to 1 second anyway.
Does it mean 10 seconds timeout wasn't enough for probe?Yes, assuming that the restart was due to Liveness.
many-animal-11139
12/23/2022, 11:59 AMgreat-jewelry-76121
12/23/2022, 1:51 PMAs I know, calico-node uses iptables to route so what calico's components attend to the datapath?The calico-node pod: • installs the CNI plugin, which networks pods when they are created, and cleans up when they are deleted • updates iptables rules on the node to implement network policy
many-animal-11139
12/23/2022, 2:49 PMgreat-jewelry-76121
12/23/2022, 2:50 PMmany-animal-11139
12/23/2022, 3:04 PM