https://rancher.com/ logo
Powered by
s

steep-family-74984

06/30/2022, 7:58 PM
Hi everybody!👋 Is anyone using multinetwork with multus in longhorn 1.3?
w

worried-businessperson-13284

07/10/2022, 10:16 PM
I'm going to try today. waiting on the second NIC for each node to arrive, going to test with a VLAN intf
it took a bit longer than I expected but I now have it working
TL;DR • installed Multus thick daemonset • created daemonset for cni-plugins, to get macvlan+dhcp working • created NetworkAttachmentDefinition for second NIC with DHCP • set LH storageNetwork to new NAD
s

steep-family-74984

07/17/2022, 6:19 AM
hi, thx for give it a try! We have facing the same delay to get it work, more time than expected. Official documentation is quite tiny for beginners 🙂 We have used macvlan to and wherabouts for ip range because we don’t have an external DHCP.
w

worried-businessperson-13284

07/17/2022, 6:30 AM
Whereabouts might work better. I'm having trouble with the dhcp server sending a default route, so the pod ends up with two equal cost default routes :(
finally got it working with macvlan+whereabouts
🙌 1
had to put the node's IP address on a macvlan sub-interface too, or iscsiadm can't talk to the containers
s

steep-family-74984

07/18/2022, 8:31 AM
yes, that’s right. And error logs are not very explicit, only the error line with nsenter help me to understand that it’s the host who initiate the connection to the container
in my lab it was effective and easy but now in a production server it’s not the same. How do you map the network namespace and the pod? Since the network namespace is never the same after in all pods and the id change after recreate (with a rolling update for example)
don’t really likee pods sharing host network
w

worried-businessperson-13284

07/25/2022, 6:44 PM
in Longhorn settings, set storageNetwork to the Multus NetworkAttachmentDefinition
you have to prefix the namespace
so mine is 
kube-system/storage-macvlan-whereabouts
then Longhorn takes care of applying that to the relevant pods
s

steep-family-74984

07/25/2022, 6:49 PM
yep, I’ve already done that, but in ns longhorn 
storage-network: longhorn-system/{{ storage_macvlan_interface_name }}
👍 1
w

worried-businessperson-13284

07/25/2022, 6:51 PM
have tried exec'ing into a pod to check addresses, routes and connectivity?
I spent a bunch of time debugging before I did that 🤦
s

steep-family-74984

07/25/2022, 6:54 PM
Which CNI do you use?
pod to pods ok pods to other nodes ok pod to underlying node nok
w

worried-businessperson-13284

07/25/2022, 6:54 PM
ah, this the macvlan thing
you need a macvlan interface on the host
s

steep-family-74984

07/25/2022, 6:55 PM
but the macvlan interface should be attached to a network namespace
w

worried-businessperson-13284

07/25/2022, 6:56 PM
not sure what you mean
I'll find the commands I used
s

steep-family-74984

07/25/2022, 6:58 PM
🙏
w

worried-businessperson-13284

07/25/2022, 6:59 PM
that's a snippet from the 
/etc/network/interfaces
file
it creates a new interface called 
shim-storage0
and links it to 
storage0
as a macvlan bridge
then puts the host's IP address for the storage network on the new interface
this way the host and the containers are all on the same bridge
s

steep-family-74984

07/25/2022, 7:02 PM
hum I’ve tried to do this with network manager, I’ve surely miss up something
w

worried-businessperson-13284

07/25/2022, 7:02 PM
oh, yeah I would disable network manager and do it manually
s

steep-family-74984

07/25/2022, 7:02 PM
thank you very much ! I’ll try it
👍 1
w

worried-businessperson-13284

07/25/2022, 7:03 PM
I disabled connmand on k3os to get this to work
btw, I renamed 
eth1
to 
storage0
, you don't have to do that
s

steep-family-74984

07/25/2022, 7:03 PM
don’t really fan of network manager but they have a pretty good ansible module
w

worried-businessperson-13284

07/25/2022, 7:04 PM
you could potentially disable Network Mangler just for the storage network interface
s

steep-family-74984

07/25/2022, 7:41 PM
thanx! it was succesfull with a mix of nmcli and ip, now I’ll automate that 🤯🔫
w

worried-businessperson-13284

07/25/2022, 7:42 PM
woohoo \0/
it was cool to watch the graphs when I pushed 15GB to an r=3 volume. only one node saw the traffic on the main NIC, all nodes saw traffic on the storage NIC
130 Views
Powered by