stocky-article-82001
11/23/2022, 12:55 PMflat-translator-32204
11/23/2022, 5:02 PMloud-lawyer-79863
11/23/2022, 5:17 PMdamp-painting-69352
11/23/2022, 7:17 PMblue-arm-71737
11/23/2022, 7:50 PMloud-lawyer-79863
11/23/2022, 8:47 PMstraight-appointment-92464
11/24/2022, 5:43 AMpolite-king-74071
11/24/2022, 1:01 PMfull-park-34540
11/24/2022, 2:58 PMinvalid bearer token, service account token has expired
Multiple pods from different namespace are in the ContainerCreating state including calico-kube-controllers and coredns in the kube-system namespace
kube-system calico-kube-controllers-54965c7ccb-rvksv 0/1 ContainerCreating 0 5h39m
kube-system coredns-d76bd69b-4dtbl 0/1 ContainerCreating 0 5h39m
kube-system kube-sriov-cni-ds-arm64-5j69h 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-5ljzh 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-5mmfz 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-5zkvh 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-8z5zc 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-bpk9q 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-gqks9 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-hsx9k 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-jrlpb 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-krt9n 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-m42j4 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-nnrw2 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-nqshk 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-q2stf 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-r8b9d 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-vn2gj 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-vstrn 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-wdbcj 0/1 ContainerCreating 0 5h38m
kube-system kube-sriov-cni-ds-arm64-zxgrl 0/1 ContainerCreating 0 5h38m
kube-system metrics-server-7cd5fcb6b7-zbkfj 0/1 ContainerCreating 0 5
kubectl describe pod throws the following error for all the pods stuck in the ContainerCreating state
Warning FailedCreatePodSandBox 55s (x1755 over 6h24m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "986af2c3af9e173a6f4084fcf73795bccf3b996c98f5a80b9f0a04a554cb8a21": plugin type="multus" name="multus-cni-network" failed (add): [cdi/cdi-apiserver-cdb4566f6-vq2zx/d82b198a-de67-4463-8e76-884e022fdc99:k8s-pod-network]: error adding container to network "k8s-pod-network": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
Journal logs flooded with the following error
Nov 24 13:00:43 dev3-kv-02 k3s[2648839]: E1124 13:00:43.020907 2648839 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"
full-park-34540
11/24/2022, 3:41 PMloud-lawyer-79863
11/24/2022, 4:00 PMERRO[2022/11/24 15:38:43] Error restoring resource grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable
ERRO[2022/11/24 15:38:45] Error restoring cluster-scoped resources [error restoring grb-lr9qh of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-lr9qh" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable error restoring grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable]
ERRO[2022/11/24 15:40:03] Error restoring cluster-scoped resources [error restoring grb-lr9qh of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-lr9qh" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable error restoring grb-cn99w of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=globalrolebindings: restoreResource: err updating resource <http://GlobalRoleBinding.management.cattle.io|GlobalRoleBinding.management.cattle.io> "grb-cn99w" is invalid: metadata.deletionGracePeriodSeconds: Invalid value: 0: field is immutable]
ERRO[2022/11/24 15:40:03] error syncing 'restore-rzrb6': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing
stocky-article-82001
11/24/2022, 5:06 PMrich-sundown-87208
11/24/2022, 6:57 PMrich-sundown-87208
11/24/2022, 6:58 PMrich-sundown-87208
11/24/2022, 6:59 PMrich-sundown-87208
11/24/2022, 7:01 PMrich-sundown-87208
11/24/2022, 7:02 PMcurl <https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s>
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: <https://curl.haxx.se/docs/sslcerts.html>
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
rich-sundown-87208
11/24/2022, 7:02 PMcurl -k <https://10.43.0.1:443/api/v1/namespaces/kube-system?timeout=45s>
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
rich-sundown-87208
11/24/2022, 7:03 PMrich-sundown-87208
11/24/2022, 8:37 PMrich-sundown-87208
11/24/2022, 8:37 PMelegant-librarian-62192
11/25/2022, 12:34 AMFATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: cannot allocate tty if runc will detach without setting console socket: unknown
The container was open with a Bash console at the point when I quit, which I suspect may be part of the problem. No problem to recreate, but curious if there's a quick way to recover from this failure mode in future.elegant-librarian-62192
11/25/2022, 2:47 AMwonderful-airplane-86139
11/25/2022, 3:47 AMsilly-jordan-81965
11/25/2022, 7:22 AMacoustic-businessperson-24722
11/25/2022, 9:03 AMsilly-jordan-81965
11/25/2022, 9:15 AMlevel=error msg="[K8s] received secret to process that was older than the last secret operated on. (256559510 vs 256559619)"
level=error msg="error syncing 'fleet-default/bootstrap-template-d4lsj-machine-plan': handler secret-watch: secret received was too old, requeuing"
salmon-portugal-38278
11/25/2022, 1:02 PMrancher-monitoring
? Ive been trying to add a PrometheusRule
using Rancher UI
following the docs here https://docs.ranchermanager.rancher.io/v2.6/how-to-guides/advanced-user-guides/monitoring-v2[…]uration-guides/advanced-configuration/prometheusrules
After creating the object, I tried to see it's really there in my prometheus dashboard. Unfort, it's not there
I looked in the rulefile
configmap that watches all the prometheusrules and it's not there too !rough-jordan-89140
11/25/2022, 2:12 PMAn empty key with operator Exists matches all keys, values and effects which means this will tolerate everything.
tolerations:
- operator: "Exists"
and
kubectl -n cattle-system patch ds/cattle-node-agent -p '{"spec":{"template":{"spec":{"tolerations":[{"operator":"Exists"}]}}}}'
do you think that the first code snippet can be added to the cattle-cluster-agent deployment manifest or it will work only with kubectl patch? We would like to make this persist with code.
Please let us know if we really can add a toleration to cattle-cluster-agent deployment manifest, thank you 🙇creamy-room-58344
11/25/2022, 3:17 PMguestinfo.metadata
base64 encoded with guestinfo.metadata.encoding: base64
as well as gzip+base64
network:
version: 2
ethernets:
nics:
match:
name: ens*
dhcp4: yes
critical: true
dhcp-identifier: mac
Testing it on vSphere directly works, but with Rancher it does not
What am I missing?