I found the webhook for rancher in the local cluster and disabled it.
It's under More resources > Admission > MutatingWebhookConfiguration and ValidatingWebhook...
You need to edit the rancher configurations and set the fail policy to Ignore.
I manually generated the certificate, but I'm sure if you try deleting the rancher webhook pod, it'll regenerate successfully.
I believe this failed because the webhook system couldn't save the new certificate due to a self referencing failure loop