Hi all, we started to have pods being unable to schedule in our Rancher cluster sometime last night. Looking into the control plane nodes the docker

kube-apiserver

container is showing a lot of errors like this:

E0213 22:18:56.420705       1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: <http://leases.coordination.k8s.io|leases.coordination.k8s.io> "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "leases" in API group "<http://coordination.k8s.io|coordination.k8s.io>" in the namespace "kube-system"

Where should I even be looking in Rancher to start troubleshooting this problem?

We also have this issue and have started looking at etcd as a potential source. Did you resolve your issue?

We resolved by tearing down the cluster and rebuilding unfortunately. Our issue looked more like a CNI driver issue than an etcd one, honestly