Hi guys! After we've upgraded the RKE2 Kuberentes guest cluster from v1.31.4 to v1.31.5 (same on v1.32.5) we've noticed that the pod networking is not working.. (with calico as container network) • we've tested also with a new cluster with v1.31.5 and the same error is happening • also, we've tested using flannel as container network and everything works as expected • when we downgrade the cluster back to v1.31.4, everything is working as expected • we are running Harvester 1.5.0 with Rancher 2.11.3 • we also have another environment with Harvester 1.4.0 and we have the same problem when we upgrade Kubernetes to a version higher than v1.31.4 So, we suspect that calico has some problems after v1.31.4 when deployed on Harvester. Any ideas on how we can solve this issue? Many thanks!

how are you upgrading? And are you talking about rke2 on VMs? sounds like it is an rke2 issue and not harvester.

Finally, I've managed to fix the issue. It was caused by Calico using the Harvester VIP LoadBalancer IP instead of the node interface. It would be good to implement a proper fix in Rancher or Harvester so that people won't encounter this problem when using LoadBalancers from Harvester. More details below: https://github.com/rancher/rke2/issues/8477#issuecomment-3024283591

interesting.

Most likely other people don't have this problem because they don't set up nginx ingress to expose it on Harvester Loadbalancer.

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      kind: Deployment
      replicaCount: 3
      ingressClassResource:
        enabled: 'true'
        default: 'true'
        name: nginx
      ingressClassByName: 'true'
      publishService:
        enabled: true
      hostPort:
        enabled: false
      service:
        enabled: true
        type: LoadBalancer
        annotations:
          <http://cloudprovider.harvesterhci.io/ipam|cloudprovider.harvesterhci.io/ipam>: "dhcp"

I had the same problem with rke2 v1.31.8 which was my first try and thought that it just wouldn't work at all. Good to know that it should work with v1.31.4 and will give that a try tomorrow