https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
p
By wire, you mean network interface?
s
yes, logical (VLANS) or physical
c
If that’s what you want to do, why are you letting clients connect directly to the nodes at all? In most scenarios, clients connect to a LoadBalancer VIP, and then only the LB is allowed to talk to nodes to expose the services. Could be an external hardware LB or cloud service, could be Kube-VIP or MetalLB running on a dedicated interface.
There is no reason that Kubernetes needs to be aware of that part at all, outside the LB implementation.
👍 1
s
So wasn't totally about the VIPs. We don't want the cluster API to be exposed to just anyone. We want that firewalled off to then limit who accesses it. Also generally the private wire needs to be in an isolated L2 so nobody can see it. Now we have noticed that the gossip communications seems to be tls encrypted now, so maybe isolation of that portion not so important.
c
Kubernetes isn’t really well designed to be run like that with complicated multi-homed nodes. Its designed to be run on cloud native infra where you have lots of subnets, generally just one or two subnets per node, and external security groups to enforce traffic restrictions. Generally you just deploy it on nodes on a fairly flat single network, MAYBE a dedicated storage network. Anything you want to expose outside the cluster you run through an external LB, and you segment the networks so that only the LBs are allowed to access the nodes. In particular, anything that relies on managing host-based firewalls or restricting things to only listening on specific interfaces is going to be fairly fussy to get working the way you want.
s
I seem to remember about 6 or 7 years back where they recommended separating out those (I think Gossip was un-encrypted then) to limit your security footprint. I've seen some documents recently that suggested the same.
Agree on services being exposed and accessed via a load balancer. It is the CAAS mgmt API I want to limit access to, not the services that K8s is exposing.
c
what do you mean by gossip? I’m not aware of anything in Kubernetes that uses a gossip protocol any more
etcd is a peer to peer cluster, vxlan is a point to point mesh with defined subnets, and so on.
s
sooo how are memberships managed then? Gossip was used in the early days for membership
c
membership in what?
s
the k8s cluster controllers. Gossip was used (from what I understood) to know when a member dropped out.
c
no. they have always used leases.
👍 1
s
and it is used in a lot of clusters for that purpose. It can be used to share limited shared data too
c
s
Thanks, I'll dig into that and reassess separating the amount of wires we are doing
So let me know what you think of this though, https://medium.com/@abhilashkrish/gossip-protocol-in-distributed-systems-b274e3aaef6f
Copy code
Kubernetes: Kubernetes utilizes the SWIM gossip protocol for managing cluster membership and health monitoring. SWIM employs a decentralized failure detection mechanism, allowing Kubernetes clusters to quickly detect and react to node failures.
Is this a difference of standard K8s and Rancher's implementation?
If so, seems like Rancher has a value add in the way they implemented 😉
Obtw the paper you gave me on leases says they are garbage collected after an hour. K8s knows the node left within fractions of seconds (which sounds more like gossip)
c
Rancher is a service that runs on top of Kubernetes.
Also this whole thing looks like it was generated by AI, there’s no value here and no reference for the claims
and the docs links just point to the landing pages for the projects
For the kubelet you can reference https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
--node-status-update-frequency duration Default: 10s
Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in Node controller.
The kubelet updates the lease on the node object every 10 seconds. if it doesn’t get renewed then the node controller knows that it has gone offline.
I think you’re working off garbage AI generated info. etcd uses RAFT for consensus across a predefined set of cluster members. Kubernetes is built on top of etcd and uses etcd leases on top of Kubernetes resources (historically on configmaps, now on a dedicated lease type) to do leader election. There is no gossip whatsoever.
s
Well there is a lot of references to k8s and to gossip for years. But may be mute as it seems whatever it is is now tls encrypted. I think we were able to accomplish in the interim the separation of ingress, API mgmt and node to node communications so we can close this out. Thanks
🤷‍♂️ 1
12 Views
Open in Slack
PreviousNext
Powered by