Hi, I am trying to install RKE2 on a centos stream 8 with 3 server nodes and 3 agent nodes...The installation is successful on the first server node but the installation fails on second and third server nodes due to the configuration mismatch error.

Jun 27 10:17:43 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[38318]: time="2022-06-27T10:17:43-04:00" level=fatal msg="starting kubernetes: preparing server: failed to validate server configuration: critical configuration value mismatch"

First Server configuration:

# BEGIN Adding RKE2 configuration
write-kubeconfig-mode: "0644"
tls-san:
- "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
node-label:
- "nodetype=master"
node-ip: "xxx.xxx.xxx.42,xxxx:xxx:x:xxx:xxx:xxxx:xxxx:aae"
cluster-cidr: "10.42.0.0/16,2001:cafe:42:0::/56"
service-cidr: "10.43.0.0/16,2001:cafe:42:1::/112"
cluster-dns: "10.43.0.10"
cluster-domain: "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
cni:
- calico
disable:
- rke2-canal
- rke2-kube-proxy
# END Adding RKE2 configuration

Second Server Configuration:

# BEGIN Adding RKE2 configuration
server: "<https://rke2-master1.xxx.xxx.xxx.42.nip.io:9345>"
token: "K10d463a80c8c1323f30fa6d97fcf91992454a43dc5c544f1c9a0de706b733b51ee::server:f6fd26cafff902300ba021b29b11eddc"
tls-san:
- "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
node-ip: "xxx.xxx.xxx.43,xxxx:xxx:x:xxx:xxx:xxxx:xxxx:5245"
cni:
- calico
disable:
- rke2-canal
- rke2-kube-proxy
# END Adding RKE2 configuration

There is no firewalld or iptables running on any of the nodes...Please help me in finding the issue here

Your Cluster DNS and Cluster Domain must be the same across servers

can you please make it some more clear...because I don't see the cluster-dns & cluster-domain fields in the secondary server node configuration

Thats the point, you need to add those 2 to the secondary server configs.

Thanks for the details...will try and come back 🙂

if you add

debug: true

to the config.yaml, it will tell you in the journalctl logs which arguments are mistmatched

I got the debug logs, but could not able to figure out the mismatch,

-- Unit rke2-server.service has begun starting up.
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> sh[77742]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> sh[77743]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[77748]: time="2022-06-27T12:49:40-04:00" level=warning msg="not running in CIS mode"
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[77748]: time="2022-06-27T12:49:40-04:00" level=info msg="Starting rke2 v1.23.6+rke2r2 (40d712e5081ac87e30e8f328f738130acf2c31f8)"
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[77748]: time="2022-06-27T12:49:40-04:00" level=info msg="Managed etcd cluster not yet initialized"
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[77748]: time="2022-06-27T12:49:40-04:00" level=debug msg="This is the server CriticalControlArgs: config.CriticalControlArgs{ClusterDNSs:[]net.IP{net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x2b, 0x0, 0xa}}, ClusterIPRanges:[]*net.IPNet{(*net.IPNet)(0xc001ebf170), (*net.IPNet)(0xc001ebf1a0)}, ClusterDNS:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x2b, 0x0, 0xa}, ClusterDomain:\"<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>\", ClusterIPRange:(*net.IPNet)(0xc001ebf1d0), DisableCCM:false, DisableHelmController:false, DisableNPC:true, DisableServiceLB:false, FlannelBackend:\"none\", FlannelIPv6Masq:false, NoCoreDNS:false, ServiceIPRange:(*net.IPNet)(0xc001ebf200), ServiceIPRanges:[]*net.IPNet{(*net.IPNet)(0xc001ebf230), (*net.IPNet)(0xc001ebf260)}}"
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[77748]: time="2022-06-27T12:49:40-04:00" level=debug msg="This is the local CriticalControlArgs: config.CriticalControlArgs{ClusterDNSs:[]net.IP{net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x2b, 0x0, 0xa}}, ClusterIPRanges:[]*net.IPNet{(*net.IPNet)(0xc00113ad20)}, ClusterDNS:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x2b, 0x0, 0xa}, ClusterDomain:\"<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>\", ClusterIPRange:(*net.IPNet)(0xc00113ad20), DisableCCM:false, DisableHelmController:false, DisableNPC:true, DisableServiceLB:false, FlannelBackend:\"none\", FlannelIPv6Masq:false, NoCoreDNS:false, ServiceIPRange:(*net.IPNet)(0xc00113ad50), ServiceIPRanges:[]*net.IPNet{(*net.IPNet)(0xc00113ad50)}}"
Jun 27 12:49:40 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[77748]: time="2022-06-27T12:49:40-04:00" level=fatal msg="starting kubernetes: preparing server: failed to validate server configuration: critical configuration value mismatch"

cluster-cidr:

Is missing from the secondary servers as well

can you please share me any reference links for secondary server node configuration

I don't really have any references for you, its all depends on your configuration. At the bare minimum, secondary servers must match the following arguments if they are present in the primary server (this is just converting the go code in the github link above into a yaml arg):

cluster-cidr
cluster-dns
cluster-domain
disable-cloud-controller
egress-selector-mode
service-cidr

Thanks much @nutritious-tomato-14686 the other server nodes are now added to the primary server node