Hey folks. We're looking at setting up a k3s cluster on a slightly unusual setup: we have two powerful machines, with 100g NICs which we can direct attach to each other (ie no switch) plus regular 10g NICs; and a number of low-powered machines each with a regular 10G NIC. All running Ubuntu. How would you go about setting this up? I'm thinking of trying to bridge a 100G and 10G NIC on one of the big machines to get everything on the same subnet to keep k3s happy. Would this work? Is there a better topology for k3s?

It depends on what type of workload you and your users plan on running on this cluster. For HA, we need 3 nodes. Perhaps, initially, take 3 of the "low-powered" machines and setup them up as control plane/etcd nodes. You can then deploy a worker on the 2 powerful machines. Many ways to do things, just depends on the need

Thanks! We don't need HA - this was more about a network topology that would work at all with k3s given the physical connection.

I'd be interested in seeing more info about that bridge arrangement and how it was done, if'n you wanted to share the results. Private info removed of course. 😄

@jolly-waitress-71272 So yes, this arrangement seemed to work fine. There's actually not much more to add. We installed 100Gb NICs in two servers and connected them with a direct attach copper cable. One of the servers then had an additional 10Gb NIC, connected to the switch to which other machines were connected. On the machine with the two NICs, we made sure the two new interfaces did not have IP addresses, created a bridge, added the two NICs to it, and assigned the bridge an IP address. The only thing that was unexpected was that we had to allow forwarding in iptables on the bridge interface (this was unexpected for us as the bridge machinery sits "below" IP in the stack).