full-park-34540
11/24/2022, 2:58 PMinvalid bearer token, service account token has expired
Multiple pods from different namespace are in the ContainerCreating state including calico-kube-controllers and coredns in the kube-system namespace
kube-system calico-kube-controllers-54965c7ccb-rvksv 0/1 ContainerCreating 0 5h39m
kube-system coredns-d76bd69b-4dtbl 0/1 ContainerCreating 0 5h39m
kube-system kube-sriov-cni-ds-arm64-5j69h 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-5ljzh 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-5mmfz 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-5zkvh 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-8z5zc 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-bpk9q 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-gqks9 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-hsx9k 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-jrlpb 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-krt9n 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-m42j4 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-nnrw2 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-nqshk 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-q2stf 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-r8b9d 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-vn2gj 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-vstrn 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-wdbcj 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-zxgrl 0/1 ContainerCreating 0 5h38m
kube-system metrics-server-7cd5fcb6b7-zbkfj 0/1 ContainerCreating 0 5
kubectl describe pod throws the following error for all the pods stuck in the ContainerCreating state
Warning FailedCreatePodSandBox 55s (x1755 over 6h24m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "986af2c3af9e173a6f4084fcf73795bccf3b996c98f5a80b9f0a04a554cb8a21": plugin type="multus" name="multus-cni-network" failed (add): [cdi/cdi-apiserver-cdb4566f6-vq2zx/d82b198a-de67-4463-8e76-884e022fdc99:k8s-pod-network]: error adding container to network "k8s-pod-network": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
Journal logs flooded with the following error
Nov 24 13:00:43 dev3-kv-02 k3s[2648839]: E1124 13:00:43.020907 2648839 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
sparse-fireman-14239
11/25/2022, 8:36 AM