Rancher v2.5.12 - all of a sudden getting “Rancher Error in driver during machine creation” when scaling with vsphere.. anyone had anything similar? it worked a few days ago, and all of a sudden -stopped?

Can you post the rancher logs?

There is really not much to go on; but here in debug mode:

2023/03/08 10:17:15 [INFO] [node-controller-rancher-machine] (k8s-em-wkr1) Adding network: /DC/network/VLAN123
2023/03/08 10:17:15 [INFO] [node-controller-rancher-machine] (k8s-em-wkr1) Resizing disk disk-1000-0 up from 52428800Kb to 53248000Kb
2023/03/08 10:17:16 [INFO] [node-controller-rancher-machine] (k8s-em-wkr1) Setting disk.enableUUID to TRUE
2023/03/08 10:17:16 [INFO] [node-controller-rancher-machine] (k8s-em-wkr1) Creating cloud-init.iso
2023/03/08 10:17:16 [INFO] [node-controller-rancher-machine] (k8s-em-wkr1) Uploading cloud-init.iso
2023/03/08 10:17:16 [DEBUG] Triggering auth refresh on u-adn6tvozol
2023/03/08 10:17:16 [DEBUG] Skipping refresh for u-adn6tvozol due to max-age
2023/03/08 10:17:16 [DEBUG] Triggering auth refresh on u-adn6tvozol
2023/03/08 10:17:16 [DEBUG] Skipping refresh for u-adn6tvozol due to max-age
2023/03/08 10:17:18 [INFO] [node-controller-rancher-machine] The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.
2023/03/08 10:17:18 [DEBUG] [GenericEncryptedStore]: set secret called for mc-m-q8ghk
2023/03/08 10:17:18 [DEBUG] [GenericEncryptedStore]: updating secret mc-m-q8ghk
2023/03/08 10:17:18 [INFO] [node-controller-rancher-machine]
2023/03/08 10:17:21 [INFO] Generating and uploading node config k8s-em-wkr1
2023/03/08 10:17:21 [DEBUG] getNodeTemplate parsed [cattle-global-nt:nt-dfxst] to ns: [cattle-global-nt] and n: [nt-dfxst]
2023/03/08 10:17:21 [DEBUG] Cleaning up [/opt/jail/c-rsw6m/management-state/node/nodes/k8s-em-wkr1]
2023/03/08 10:17:21 [ERROR] error syncing 'c-rsw6m/m-q8ghk': handler node-controller: Error creating machine: Error in driver during machine creation: 500 Internal Server Error, requeuing

Can you enable trace logging? And is this an rke1 cluster?

RKE1, correct.

Last soap call before it logs the error

Hi David, our server dept found something in the vSphere logs, that pointed to some crypto permission that was missing. When re-enabled, this worked again. Thanks for the support