Hi, can anyone help me how to safely fix this error on one of the three prod rancher nodes? ,"msg":"request sent was ignored by remote peer due to cluster ID mismatch", However my rancher cluster is up and functioning . Not sure if the quorum is 2 for 3 nodes deployment.

By rancher nodes, do you mean the servers offering the rancher service or some cluster that is supported? In either case, can't you remove the server from the cluster and do a fresh import?

Not sure if the quorum is 2 for 3 nodes deployment.

kubectl get nodes -owide

should help show that if this is at the rke/k3s level. It might be helpful if you specify where exactly you're seeing that message.

@best-appointment-66450 By rancher node yes i mean the servers offering the rancher services,. One of the etcd server in a nodes was not able to to communicate with other two because of cluster id mismatch .

I've got etcd on a separate MariaDB cluster with Galera. That's a separate issue to Rancher, but could be dealt with in much the same way, restarting the service on whichever server was showing cluster ID mismatch. Of course, I have other problems with that: I initially set up Rancher with just a single etcd server and haven't found an easy way to get Rancher to use all the servers of the cluster.