When setting up a nginx or haproxy LB, do I set that up outside of kubernetes environment? Like as a docker service on the main rancher docker server? In tandem?

You can have it a number of ways. The setup I prefer is to have an ingress controller (nginx ingress controller) deployed on the cluster. The ingress controller service is configured as NodePort (that exposes the ingress controller on the worker node ports). Outside the cluster I configure a LB cluster, that has it's backend configured for the worker nodes.

So i'm looking for documentation or a blog/write-up on how to set this up.

Here, you can substitute "Public Endpoint" with a load balancer. https://docs.nginx.com/nginx-ingress-controller/intro/how-nginx-ingress-controller-works/ Of course, there are other ways to do this.

That would work. This is all in my homelab.