Having some issues with overlay networking in RKE2... I went through the steps in here: https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/networking My firewall configs: Server nodes:

# firewall-cmd --zone=public --list-all
public (default, active)
  target: default
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: ens192
  sources:
  services: etcd-client etcd-server kube-apiserver kubelet wireguard
  ports: 9345/tcp 9099/tcp 30000-32767/tcp 2381/tcp 51821/udp 8472/udp
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Agent nodes:

# firewall-cmd --zone=public --list-all
public (default, active)
  target: default
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: ens192 ens224
  sources:
  services: kubelet wireguard
  ports: 9099/tcp 30000-32767/tcp 8472/udp 51821/udp
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Output from overlaytest:

# ./overlaytest.sh 
=> Start network overlay test
k8sagent02 can reach k8sagent02
command terminated with exit code 1
FAIL: overlaytest-4dtr4 on k8sagent02 cannot reach pod IP 10.252.2.2 on k8ssvr02
command terminated with exit code 1
FAIL: overlaytest-4dtr4 on k8sagent02 cannot reach pod IP 10.252.0.4 on k8ssvr01
command terminated with exit code 1
FAIL: overlaytest-4dtr4 on k8sagent02 cannot reach pod IP 10.252.3.19 on k8sagent01
command terminated with exit code 1
FAIL: overlaytest-4dtr4 on k8sagent02 cannot reach pod IP 10.252.1.2 on k8ssvr03
command terminated with exit code 1
FAIL: overlaytest-8vxld on k8ssvr02 cannot reach pod IP 10.252.4.3 on k8sagent02
k8ssvr02 can reach k8ssvr02
command terminated with exit code 1
FAIL: overlaytest-8vxld on k8ssvr02 cannot reach pod IP 10.252.0.4 on k8ssvr01
command terminated with exit code 1
FAIL: overlaytest-8vxld on k8ssvr02 cannot reach pod IP 10.252.3.19 on k8sagent01
command terminated with exit code 1
FAIL: overlaytest-8vxld on k8ssvr02 cannot reach pod IP 10.252.1.2 on k8ssvr03
command terminated with exit code 1
FAIL: overlaytest-ds7sh on k8ssvr01 cannot reach pod IP 10.252.4.3 on k8sagent02
command terminated with exit code 1
FAIL: overlaytest-ds7sh on k8ssvr01 cannot reach pod IP 10.252.2.2 on k8ssvr02
k8ssvr01 can reach k8ssvr01
command terminated with exit code 1
FAIL: overlaytest-ds7sh on k8ssvr01 cannot reach pod IP 10.252.3.19 on k8sagent01
command terminated with exit code 1
FAIL: overlaytest-ds7sh on k8ssvr01 cannot reach pod IP 10.252.1.2 on k8ssvr03
command terminated with exit code 1
FAIL: overlaytest-jw99g on k8sagent01 cannot reach pod IP 10.252.4.3 on k8sagent02
command terminated with exit code 1
FAIL: overlaytest-jw99g on k8sagent01 cannot reach pod IP 10.252.2.2 on k8ssvr02
command terminated with exit code 1
FAIL: overlaytest-jw99g on k8sagent01 cannot reach pod IP 10.252.0.4 on k8ssvr01
k8sagent01 can reach k8sagent01
command terminated with exit code 1
FAIL: overlaytest-jw99g on k8sagent01 cannot reach pod IP 10.252.1.2 on k8ssvr03
command terminated with exit code 1
FAIL: overlaytest-mmsv9 on k8ssvr03 cannot reach pod IP 10.252.4.3 on k8sagent02
command terminated with exit code 1
FAIL: overlaytest-mmsv9 on k8ssvr03 cannot reach pod IP 10.252.2.2 on k8ssvr02
command terminated with exit code 1
FAIL: overlaytest-mmsv9 on k8ssvr03 cannot reach pod IP 10.252.0.4 on k8ssvr01
command terminated with exit code 1
FAIL: overlaytest-mmsv9 on k8ssvr03 cannot reach pod IP 10.252.3.19 on k8sagent01
k8ssvr03 can reach k8ssvr03
=> End network overlay test

I've also excluded the various tunnel interfaces from NetworkManager per this https://docs.rke2.io/known_issues#networkmanager

# cat /etc/NetworkManager/conf.d/rke2-canal.conf
[keyfile]
unmanaged-devices=interface-name:flannel*;interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali

How do I begin troubleshooting this? I'm running rke2 stable on AlmaLinux 10

technically we don’t support running with firewalld/ufw enabled: https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking

Ah, yeah looks like the case. I thought I only read ufw not firewalld, not sure how I missed that.

@creamy-pencil-82913 How do we get around the requirement for firewalld being active on DoD STIG-mandated systems? Obviously deviations can be approved like this, but you know how compliance people can be. Would it be possible to add it to the RKE2 STIG as a requirement or aspect? Something that I can point to my compliance officer and state, "Look, it's here in the STIG for this application."

idk, you’d have to ask the Rancher Fed folks about that. We don’t deal with STIGs or anythig else relating to government/military hardening requirements. All we do in RKE2 itself is CIS Benchmark

Thanks!