Huh, I tried added allowing ingress:

- from:
  - ipBlock:
      cidr: <my cluster IP block>

and it still didn't help 🤔

Are you externally health-checking the pods or something?

or do your health checks depend on something that you’ve blocked?

As soon as I enable the network policies, I get this:

Warning  Unhealthy         2m41s (x3 over 3m1s)   kubelet            Liveness probe failed: Get "http://[fda5:f00d:c:0:2::2c6]:80/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy         2m41s (x3 over 3m1s)   kubelet            Readiness probe failed: Get "http://[fda5:f00d:c:0:2::2c6]:80/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

It's just the standard probes

There are two network policies in question, here: a namespace wide default-deny on both ingress and egress, and then one that allows traffic from the ingress controllers for these particular pods. I'm able to access the pods via the latter, but the probes fail and the pod is restarted

afaik netpols don’t affect health checks but perhaps that is CNI specific

I'll try to create a minimum reproducer, see if you can duplicate

Okay here you go @creamy-pencil-82913 :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        livenessProbe: &probe
          httpGet:
            path: /
            port: 80
        readinessProbe:
          <<: *probe
        startupProbe:
          <<: *probe
---
apiVersion: v1
kind: Service
metadata:
  name: ngnix-service
spec:
  selector:
    app: nginx
  ports:
  - port: 80
---
kind: NetworkPolicy
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
kind: NetworkPolicy
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
metadata:
  name: nginx
spec:
  policyTypes:
    - Ingress
    - Egress
  podSelector:
    matchLabels:
      app: nginx
  ingress:
    # Allow access from ingress controller (i.e. from the internet)
    - from:
      - namespaceSelector:
          matchLabels:
            name: nginx-ingress
        podSelector:
          matchLabels:
            <http://app.kubernetes.io/name|app.kubernetes.io/name>: nginx-ingress

  egress:
    # Allow access to DNS
    - to:
      - namespaceSelector:
          matchLabels:
            <http://kubernetes.io/metadata.name|kubernetes.io/metadata.name>: kube-system
        podSelector:
          matchLabels:
            k8s-app: kube-dns
      ports:
      - protocol: UDP
        port: 53

Interestingly, if I leave only the default-deny network policy, the probes appear to work fine

But it's the addition of the "nginx" policy, which is intended to open things up a little, that makes them start to fail

Am I doing something weird in that policy?

I wonder if it could be related to https://github.com/k3s-io/k3s/issues/9925

Are you able to try with this week’s releases?

ipv6 is the primary in this cluster, so to the length of my knowledge I can either do dual stack or only ipv6, but not ipv4, and probes always use ipv6 either way

Those issues sound awfully related to the mac-address-changing bug I logged last week, in that those packets NOT being dropped in the past would explain why the mac changing was never a problem

Sadly, this is a prod cluster, I'm not all that comfortable using a non-stable release on it

Interesting data point: I was wrong about the default-deny policy. But the policy needs to be in place before the pod fires up

If I put it in place after, the pod is fine until it restarts

Then the probes start to fail

@creamy-pencil-82913 I went ahead and logged https://github.com/k3s-io/k3s/issues/10030 to make async easier