This is a production system, our RKE cluster is still running but who knows for how long, I'm a bit concerned that I can't get to the UI and the local cluster isn't in the best shape
Thanks, I actually saw that yesterday and chased that around for a while without any luck. I think I just figured out my problem though, it was the upgrade from k3s 1.20 to 1.23 and the conflict with traefik 1 vs 2. We upgraded k3s a couple weeks ago and everything worked fine, I guess the restart of all of our instances on Sunday brought out the issues. Thanks to @creamy-pencil-82913 comment here I'm back in business
It was hard to track down because there's random errors in pod logs in all of our clusters, so it took a while to figure out what was actually causing a problem