Hi everybody wave Is anyone using multinetwork with multus i rancher-users #longhorn-storage

Docs

Join the conversation Join Slack

Channels

Title

steep-family-74984

06/30/2022, 7:58 PM

Hi everybody!👋 Is anyone using multinetwork with multus in longhorn 1.3?

worried-businessperson-13284

07/10/2022, 10:16 PM

I'm going to try today. waiting on the second NIC for each node to arrive, going to test with a VLAN intf

it took a bit longer than I expected but I now have it working

https://github.com/deekue/homelab/tree/main/cluster-critical/multus

https://github.com/deekue/homelab/blob/ad8448b2489e702601284a41b755b65f8d9cbba0/cluster-critical/longhorn-system/longhorn-helmchart.yaml#L31

https://github.com/deekue/homelab/blob/main/cluster-critical/multus/nads/storage-macvlan.yaml

TL;DR • installed Multus thick daemonset • created daemonset for cni-plugins, to get macvlan+dhcp working • created NetworkAttachmentDefinition for second NIC with DHCP • set LH storageNetwork to new NAD

steep-family-74984

07/17/2022, 6:19 AM

hi, thx for give it a try! We have facing the same delay to get it work, more time than expected. Official documentation is quite tiny for beginners 🙂 We have used macvlan to and wherabouts for ip range because we don’t have an external DHCP.

worried-businessperson-13284

07/17/2022, 6:30 AM

Whereabouts might work better. I'm having trouble with the dhcp server sending a default route, so the pod ends up with two equal cost default routes :(

finally got it working with macvlan+whereabouts

🙌 1

had to put the node's IP address on a macvlan sub-interface too, or iscsiadm can't talk to the containers

steep-family-74984

07/18/2022, 8:31 AM

yes, that’s right. And error logs are not very explicit, only the error line with nsenter help me to understand that it’s the host who initiate the connection to the container

in my lab it was effective and easy but now in a production server it’s not the same. How do you map the network namespace and the pod? Since the network namespace is never the same after in all pods and the id change after recreate (with a rolling update for example)

don’t really likee pods sharing host network

worried-businessperson-13284

07/25/2022, 6:44 PM

in Longhorn settings, set storageNetwork to the Multus NetworkAttachmentDefinition

you have to prefix the namespace

so mine is

kube-system/storage-macvlan-whereabouts

which is defined here https://github.com/deekue/homelab/blob/main/cluster-critical/multus/nads/storage-macvlan-whereabouts.yaml

then Longhorn takes care of applying that to the relevant pods

steep-family-74984

07/25/2022, 6:49 PM

yep, I’ve already done that, but in ns longhorn

storage-network: longhorn-system/{{ storage_macvlan_interface_name }}

👍 1

worried-businessperson-13284

07/25/2022, 6:51 PM

have tried exec'ing into a pod to check addresses, routes and connectivity?

I spent a bunch of time debugging before I did that 🤦

steep-family-74984

07/25/2022, 6:54 PM

Which CNI do you use?

pod to pods ok pods to other nodes ok pod to underlying node nok

worried-businessperson-13284

07/25/2022, 6:54 PM

ah, this the macvlan thing

you need a macvlan interface on the host

steep-family-74984

07/25/2022, 6:55 PM

but the macvlan interface should be attached to a network namespace

worried-businessperson-13284

07/25/2022, 6:56 PM

not sure what you mean

I'll find the commands I used

steep-family-74984

07/25/2022, 6:58 PM

🙏

worried-businessperson-13284

07/25/2022, 6:59 PM

https://github.com/deekue/homelab/blob/24e7c0f77b2f5948a6d84a8ab2009fab2f75cbe7/cluster-init/nodes/mknode.sh#L67-L74

that's a snippet from the

/etc/network/interfaces

file

it creates a new interface called

shim-storage0

and links it to

storage0

as a macvlan bridge

then puts the host's IP address for the storage network on the new interface

this way the host and the containers are all on the same bridge

steep-family-74984

07/25/2022, 7:02 PM

hum I’ve tried to do this with network manager, I’ve surely miss up something

worried-businessperson-13284

07/25/2022, 7:02 PM

oh, yeah I would disable network manager and do it manually

steep-family-74984

07/25/2022, 7:02 PM

thank you very much ! I’ll try it

👍 1

worried-businessperson-13284

07/25/2022, 7:03 PM

I disabled connmand on k3os to get this to work

btw, I renamed

eth1

storage0

, you don't have to do that

steep-family-74984

07/25/2022, 7:03 PM

don’t really fan of network manager but they have a pretty good ansible module

worried-businessperson-13284

07/25/2022, 7:04 PM

you could potentially disable Network Mangler just for the storage network interface

steep-family-74984

07/25/2022, 7:41 PM

thanx! it was succesfull with a mix of nmcli and ip, now I’ll automate that 🤯🔫

worried-businessperson-13284

07/25/2022, 7:42 PM

woohoo \0/

it was cool to watch the graphs when I pushed 15GB to an r=3 volume. only one node saw the traffic on the main NIC, all nodes saw traffic on the storage NIC

View count: 33

Back to #longhorn-storage Join thread in Slack