https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
l
Wondering if maybe the networkpolicy engine here can grow support for mapping a particular name/namespace or label back to the proper addresses even though they aren't represented by k8s objects
c
I think you’re confusing things… are you talking about 
kube-proxy
, or 
kubectl proxy
?
l
Almost certainly 😆 . I don't actually know, but given that this is a kubectl plugin I suspect the latter
c
yes, that goes through the apiserver, and the apiserver runs on a node not inside the cluster - so from the CNI perspective you will see the source of the connection as a node IP, not a pod IP.
l
Is that the same as if I run 
kubectl get --raw "/api/v1/namespaces/<namespace>/pods/<database pod>:8000/proxy/readyz"
?
Because that gives me a log entry on my database pod saying the traffic is coming from my cluster network, not my node network
And using my node IPs in the network policy didn't work, but the cluster CIDR did
c
not exactly - kubectl proxy exposes the whole apiserver on a local port. the node and pod proxy subresources are different, but they do also go through the apiserver
l
Oh this is helpful, thank you, I indeed was swimming in proxies
c
using my node IPs in the network policy didn’t work, but the cluster CIDR did
Yeah so that’ll be CNI specific. They each do it a little differently. With flannel you’ll see the .0 address in the node’s pod cidr as the source of the traffic. Other CNIs may show a node IP, or something else. There is no standard behavior here.
l
Yes okay, that's exactly what I'm seeing, the .0 address (and yeah I'm just using flannel)
Darn, that's hard to write a policy for
Each node gets a non-deterministic slice of the cluster network
But I suppose the network policy lib used here needs to not care about which CNI is being used huh
c
well it is assigned sequentially, and in blocks of a known size - /24 by default
l
So it can't magically support this
Yeah but I can't do that dynamically from a network policy, of course
c
and yeah, you will need to generate them on your own. you can’t just say “all nodes” in the network policy itself. You’d need to have a controller do it.
l
This isn't the first time I've hit this kind of fallout from the API server not being in the cluster. CNPG pods also need to send their status back via the API server, which means I have my node CIDR hard-coded for egress (not the pod network, which is interesting). Would really be nice to figure out how to make that nicer
c
on rke2 in CIS mode we actually do run a controller that does something like this - but it is only necessary because RKE2 in hardened mode applies default policies that block things. K3s doesn’t do this. https://github.com/rancher/rke2/blob/master/pkg/controllers/cisnetworkpolicy/controller.go
l
Interesting idea. I know there are controllers that support DNS, too, maybe it's time to investigate some, heh
(and yes, I also apply default deny-all policies)
c
The apiserver is NEVER in the cluster. Not anywhere. The control-plane always operates outside the cluster network. In some cloud-provider networks where the pod IPs are assigned from the cloud’s VPC network for example, it may have a load-balancer endpoint on the same network, but it will never be “in” the cluster.
you might see also https://kubernetes.io/docs/concepts/services-networking/network-policies/#what-you-can-t-do-with-network-policies-at-least-not-yet > • Node specific policies (you can use CIDR notation for these, but you cannot target nodes by their Kubernetes identities specifically).
l
Okay you clearly know more than I do, but what does it mean when I see other folks online have kube-apiserver stuff in their kube-system namespace?
I assumed that meant other k8s installations ran them in pods (and thus could use standard selectors in network policies), but I believe you when you say that's not the case
c
… those are mirror pods
apiserver runs as static pods with host network. the manifests are managed locally on the node, the mirror pods just show up in 
get pod
so that you can see them and their resources are accounted for. you can’t actually do anything with them remotely.
l
Ahhhh, I've never heard of this before
c
l
Yeah I'm reading that now
Okay interesting-- I don't see that on my k3s installation though. Does that mean in k3s they aren't static pods, but something else altogether?
c
K3s runs everything in a single process. kubelet, apiserver, etcd, scheduler, controller-manager, all of it.
that is how it manages to be so lightweight
l
Gotcha. In systems where the apiserver is a static pod, can one use the mirror pods' namespace/name/labels in a networkpolicy? Or when you say "you can't actually do anything with them" is that part of what you mean?
c
no, because it doesn’t have its own IP. Like I said it runs with host network - because it needs to come up before the CNI is deployed.
l
Okay fascinating, thank you
c
Otherwise it would be a chicken-and-egg thing. CNI needs to know what IPs to use for pods. That comes from podCIDR field on the node. Node is a resource that is stored in the apiserver. but apiserver isn’t up yet because it doesn’t have an IP from the CNI…
l
Yeah that makes sense. I'm reading now about how other CNIs (reading about calico specifically) actually had to introduce new concepts to cover this kind of thing
Definitely some sharp edges on vanilla network policies that I need to consider
b
We use Cilium with BGP and Cilium Egress Gateway so that we can have network policies that have a better "understanding" of where the request has come from rather than just "one of the nodes". Someone else in the team, far cleverer than I (which actually covers all of them) set it up. It took a while.
l
Yeah @best-appointment-66450 that's pretty much the conclusion I've come to as well. One of those "when I have time" things 🙂
32 Views
Open in Slack
PreviousNext
Powered by