This message was deleted.

how many control plane node you have ?

3 Control plane nodes, but none of them is running now

you are getting this erro when ? scaling up ? can you explain the scenario

I edited the cluster config to increase the node CPU and Memory, then new nodes were stuck in "Waiting for init node". I deleted those nodes and after that it started throwing the above error.

oh okay got it so what happens is when you deploy control plane nodes ( odd numbers ) because of RAF algo that etcd uses so that means the etcd main conf was on one of those nodes and when it got deleted ( when it was stuck at the init node ) that conf got deleted as well that was used by etcd . Now i am assuming you deleted all the three nodes and they started showing this error , because now the nodes think and try to find the existing etcd plane but thats not available now so its asking for a backup to restore the etcd

And as no backup is available so it's impossible to restore it, right? What do you suggest to avoid this situation in future? How many number of nodes should I select considering the application load was well handled by 3 nodes of 16 GB and 8vCPUs.

well to be honest its better to segregate the roles so that you have etcd seperately that way etcd will not get affected if you change any config on the masters node , second thing is when provisioning a cluster you can take backup of your etcds so that atleast you will sort of have a state with you to go back to incase of errors also which platform you are using to provision this cluster on ?

Running on Vsphere, and running a different etcd is currently not possible.

okay and how are you assigning ips to vm ? i am assuming you are using vapps ?

Yes, it is automatically taking IPs from the network pool

my set up is also same with three master + etcd role nodes i have enabled the backup for etcd

Where are you storing the backups?

well as of now locally only

So currently I can go with the odd number of nodes, right? And taking the regular backup of etcd will save from such problems in future.

yes bro also when you make changes in the config of control plane node never delete all the nodes at once as it will remove the etcd config plane and you will face the same error

Ok, thanks a lot for the help. Any idea what's the reason for the node getting stuck in init?

well it gets stuck in the init phase when either it doesnt get proper resources , seems more like a vsphere related error to me . coz when rancher shows " waiting for agent to apply initial plan " thats where vcenter provides network to the node using the cloud init file

How can I go with segregating etcd nodes on Vsphere? Any suggestions?

bro you have to create 3 seperate pools for 3 roles , master worker and etcd

Then how to connect them all? Will it be automatically done?

yes bro they connect automatically

Perfect, I will give it a try. Thanks 👍

yeah 😄