This is a production system, our RKE cluster is still running but who knows for how long, I'm a bit concerned that I can't get to the UI and the local cluster isn't in the best shape

Not too familiar with this issue itself but this seems like it could be related: https://github.com/rancher/rancher/issues/37039

Thanks, I actually saw that yesterday and chased that around for a while without any luck. I think I just figured out my problem though, it was the upgrade from k3s 1.20 to 1.23 and the conflict with traefik 1 vs 2. We upgraded k3s a couple weeks ago and everything worked fine, I guess the restart of all of our instances on Sunday brought out the issues. Thanks to @creamy-pencil-82913 comment here I'm back in business