https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
e
I used the NetApp KB to test the connectivity from a test pod and confirmed it cant reach outside itself. The GW IP shows up as an APIPA IP in side the pod.
I have a few production clusters I have setup with RKE2/Calico/RHEL in an identical setup and I have not had any issues with the inter-node comms. I made sure all my sysctl settings for ip_forwarding were set to 1, which was usually the issue when I did have issues.
Copy code
NAME                                  READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
curl-pod                              1/1     Running   2 (9m40s ago)    29m     10.42.242.212   csi-test1   <none>           <none>
trident-controller-75ccc7c5f5-s9cs5   6/6     Running   0                3h15m   10.42.199.4     csi-test3   <none>           <none>
trident-node-linux-8tr66              1/2     Running   39 (5m30s ago)   3h15m   10.7.54.52      csi-test2   <none>           <none>
trident-node-linux-cx5w9              2/2     Running   2 (3h13m ago)    3h15m   10.7.54.53      csi-test3   <none>           <none>
trident-node-linux-p6kvl              1/2     Running   39 (5m38s ago)   3h15m   10.7.54.51      csi-test1   <none>           <none>
Looking a bit closer I see the Ip assigned to the trident nodes are the ones assigned to the k8s nodes, not the internal IP's, so maybe thats where it falls down. My production clusters use longhorn, so cant really compare.
Copy code
# kubectl get ippool default-ipv4-ippool -o yaml
apiVersion: <http://crd.projectcalico.org/v1|crd.projectcalico.org/v1>
kind: IPPool
metadata:
  creationTimestamp: "2025-09-18T05:50:54Z"
  generation: 1
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: tigera-operator
  name: default-ipv4-ippool
  resourceVersion: "920"
  uid: 220fc13b-6c52-4048-9edd-46aa7fbc1843
spec:
  allowedUses:
  - Workload
  - Tunnel
  assignmentMode: Automatic
  blockSize: 26
  cidr: 10.42.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always
I dont see too much different between the ippool above and one that works on my production cluster (accounting for v1.29 to v1.31 comparison)
Copy code
# k get nodes -o wide
NAME        STATUS   ROLES                       AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                KERNEL-VERSION                  CONTAINER-RUNTIME
csi-test1   Ready    control-plane,etcd,master   12d   v1.31.12+rke2r1   10.7.54.51    <none>        Red Hat Enterprise Linux 8.10 (Ootpa)   4.18.0-553.70.1.el8_10.x86_64   <containerd://2.0.5-k3s2>
csi-test2   Ready    <none>                      12d   v1.31.12+rke2r1   10.7.54.52    <none>        Red Hat Enterprise Linux 8.10 (Ootpa)   4.18.0-553.70.1.el8_10.x86_64   <containerd://2.0.5-k3s2>
csi-test3   Ready    <none>                      12d   v1.31.12+rke2r1   10.7.54.53    <none>        Red Hat Enterprise Linux 8.10 (Ootpa)   4.18.0-553.70.1.el8_10.x86_64   <containerd://2.0.5-k3s2>
nodes seem happy
The curl-pod (test pod) can ping the external IPs (10.7.54.x) but cant ping the trident controller. So not sure why it got a correct private IP and the others did not. Assuming they were supposed to get 10.42.x.x IP's which Im not even sure of.
Tried setting the felixconfiguration defaultEndpointToHostAction to Accept instead of drop and that did nothing to help.
Got confirmation the trident pods are supposed to be getting the external IP, so all that looks good.
No networkPolicy or GlobalNetworkPolicy are defined in any namespace, which is supposed to "allow all"
well.. damn
turns out it was having a 2nd NIC on the server
as soon as I disconnected that NIC from each host in vCenter, things all synced up
I kind of need that NIC as I want it to be dedicated to the NFS traffic on its own VLAN, but at least I know what the current issue is.
12 Views
Open in Slack
PreviousNext
Powered by