I ve got a problem with calico on one of my RKE2 clusters 4 Rancher Users #rke2

I've got a problem with calico on one of my RKE2 c...
crooked-cat-21365
01/08/2024, 11:35 AM
I've got a problem with calico on one of my RKE2 clusters: 4 of 5 hosts say "Waiting for probes: calico" in Rancher. The calico node pods say something like
Copy code
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  120m  default-scheduler  Successfully assigned calico-system/calico-node-pvlqb to <http://srvl043.ac.aixigo.de|srvl043.ac.aixigo.de>
  Normal   Pulled     120m  kubelet            Container image "<http://docker.io/rancher/mirrored-calico-pod2daemon-flexvol:v3.26.3|docker.io/rancher/mirrored-calico-pod2daemon-flexvol:v3.26.3>" already present on machine
  Normal   Created    120m  kubelet            Created container flexvol-driver
  Normal   Started    120m  kubelet            Started container flexvol-driver
  Normal   Pulled     120m  kubelet            Container image "<http://docker.io/rancher/mirrored-calico-cni:v3.26.3|docker.io/rancher/mirrored-calico-cni:v3.26.3>" already present on machine
  Normal   Created    120m  kubelet            Created container install-cni
  Normal   Started    120m  kubelet            Started container install-cni
  Normal   Pulled     120m  kubelet            Container image "<http://docker.io/rancher/mirrored-calico-node:v3.26.3|docker.io/rancher/mirrored-calico-node:v3.26.3>" already present on machine
  Normal   Created    120m  kubelet            Created container calico-node
  Normal   Started    120m  kubelet            Started container calico-node
  Warning  Unhealthy  120m  kubelet            Readiness probe failed: calico/node is not ready: felix is not ready: Get "<http://localhost:9099/readiness>": dial tcp [::1]:9099: connect: connection refused
W0108 08:33:20.021173      54 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  113m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:40:14.360305     889 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  113m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:40:44.359313    1904 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  112m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:41:14.390890    2853 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  112m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:41:14.542028    2875 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  112m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:41:44.371104    3855 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  112m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:41:51.594067    4127 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  111m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:42:14.359380    4831 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  111m  kubelet  Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:42:27.574962    4852 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  Unhealthy  109m (x9 over 111m)  kubelet  (combined from similar events): Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
W0108 08:45:09.570800   10229 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
  Warning  FailedMount  101m               kubelet  MountVolume.SetUp failed for volume "node-certs" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  101m               kubelet  MountVolume.SetUp failed for volume "tigera-ca-bundle" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  70m                kubelet  MountVolume.SetUp failed for volume "tigera-ca-bundle" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  70m                kubelet  MountVolume.SetUp failed for volume "node-certs" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  69m (x4 over 70m)  kubelet  MountVolume.SetUp failed for volume "kube-api-access-tznkj" : failed to fetch token: Post "<https://127.0.0.1:6443/api/v1/namespaces/calico-system/serviceaccounts/calico-node/token>": dial tcp 127.0.0.1:6443: connect: connection refused
  Warning  FailedMount  43m                kubelet  MountVolume.SetUp failed for volume "tigera-ca-bundle" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  43m                kubelet  MountVolume.SetUp failed for volume "node-certs" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  42m (x4 over 43m)  kubelet  MountVolume.SetUp failed for volume "kube-api-access-tznkj" : failed to fetch token: Post "<https://127.0.0.1:6443/api/v1/namespaces/calico-system/serviceaccounts/calico-node/token>": dial tcp 127.0.0.1:6443: connect: connection refused
  Warning  FailedMount  16m                kubelet  MountVolume.SetUp failed for volume "node-certs" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  16m                kubelet  MountVolume.SetUp failed for volume "tigera-ca-bundle" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  15m (x4 over 16m)  kubelet  MountVolume.SetUp failed for volume "kube-api-access-tznkj" : failed to fetch token: Post "<https://127.0.0.1:6443/api/v1/namespaces/calico-system/serviceaccounts/calico-node/token>": dial tcp 127.0.0.1:6443: connect: connection refused
  Warning  FailedMount  15m                kubelet  MountVolume.SetUp failed for volume "kube-api-access-tznkj" : failed to fetch token: serviceaccounts "calico-node" is forbidden: User "system:node:<http://srvl043.ac.aixigo.de|srvl043.ac.aixigo.de>" cannot create resource "serviceaccounts/token" in API group "" in the namespace "calico-system": no relationship found between node '<http://srvl043.ac.aixigo.de|srvl043.ac.aixigo.de>' and this object
Does this ring a bell? Every helpful hint would be highly appreciated. Environment: Debian 12, Rancher 2.7.9, RKE2 1.26.11
28 Views
Open in Slack
Previous Next