Quick question: is there any concrete downside to using data locality: best-effort? Assume the workloads are running on a dedicated server. If the server dies and the pod gets rescheduled it will just use another replica, correct?

Only downside is if you were using something like bin packing to fill up scheduled pods on one node before the others. Even then, as long as you have other replicas on other nodes, it should still be ok.