Hi all, my k3s cluster in prod is in a bad state. We run Rancher in a 3 node k3s cluster on ec2 instances using RDS as an external datastore. I came in this morning to Rancher being down, all 3 of the rancher pods were on the same node, all with 0/1 in the Ready column. I rebooted the node, which didn't help. I noticed local-path-provisioner, fleet-agent, and fleet-controller all have hundreds of restarts. The errors in the local-path-provisioner log are like (sorry, air-gapped environment so manual transcription)

leaderelection.go Failed to update lock: Put <https://10.43.0.1:443/api/v1/namespaces/kube-system/endpoints/rancher.io-local-path:EOF>
leaderelection.go failed to renew lease kube-system/rancher.io-local-path: timed out waiting for condition

I tried deleting that pod so it'd start fresh but it hasn't tried to restart yet. I did that first in a test environment and it started right back up. Looking at the fleet-agent logs I see this at then end

error syncing 'cluster-fleet-local.... : handler bundle-deploy: another operation (install/upgrade/rollback) is in progress
error syncing cluster-fleet-local... |handler bundle-deploy: contents.fleet.cattle.io "<long-string>" is forbidden: User system:serviceaccount:cluster-fleet-... : request... cannot get rsource contents in API group fleet.cattle.io at the cluster scope, requeing
leaderelection.go Failed to update lock: configmaps "fleet-agent-lock" is forbidden : User "system:serviceaccount:cattle-fleet-local-system:fleet-agent" cannot update resource "configmaps" in API group "" in the namespace "cattle-fleet-local-system"

Not quite sure where to go from here. I added a new k3s management node and it joined fine, but no pods are running on it