AFAIK adding or removing masters from k8s is not an easy task to afford.
Considering you´re moving from physicals to VMs, how about create a new controlplane on the virtual nodes, adding workers and perform a blue/green migration?
Moving the core components of k8s should involve etcd data migration, controller and components along with losing both etcd/controlPlane quorum, not even talking about the certificates issues when you try to do something like move the components.