Is it going to break my system if I run Rancher in a VM inside harvester and then use that Rancher instance to manage harvester?

For testing/development you should be okay but don't do this if production - see https://docs.harvesterhci.io/v1.0/rancher/rancher-integration/#deploying-rancher-server

What issues could crop up if I did use it in prod?

For production you should run Rancher in a separate, multi-node cluster. If you run Rancher within Harvester then if anything happens to Harvester (which you might do by mistake through Rancher) you've also stuffed up Rancher hence okay for test/dev but not production.

If I only have one server how would I do this?

Potentially - 3 nodes are recommended so anywhere (virtual or physical) that isn't within Harvester.