https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# longhorn-storage
a
This message was deleted.
w
I'm going to try today. waiting on the second NIC for each node to arrive, going to test with a VLAN intf
it took a bit longer than I expected but I now have it working
TL;DR • installed Multus thick daemonset • created daemonset for cni-plugins, to get macvlan+dhcp working • created NetworkAttachmentDefinition for second NIC with DHCP • set LH storageNetwork to new NAD
s
hi, thx for give it a try! We have facing the same delay to get it work, more time than expected. Official documentation is quite tiny for beginners 🙂 We have used macvlan to and wherabouts for ip range because we don’t have an external DHCP.
w
Whereabouts might work better. I'm having trouble with the dhcp server sending a default route, so the pod ends up with two equal cost default routes :(
finally got it working with macvlan+whereabouts
🙌 1
had to put the node's IP address on a macvlan sub-interface too, or iscsiadm can't talk to the containers
s
yes, that’s right. And error logs are not very explicit, only the error line with nsenter help me to understand that it’s the host who initiate the connection to the container
in my lab it was effective and easy but now in a production server it’s not the same. How do you map the network namespace and the pod? Since the network namespace is never the same after in all pods and the id change after recreate (with a rolling update for example)
don’t really likee pods sharing host network
w
in Longhorn settings, set storageNetwork to the Multus NetworkAttachmentDefinition
you have to prefix the namespace
so mine is 
kube-system/storage-macvlan-whereabouts
then Longhorn takes care of applying that to the relevant pods
s
yep, I’ve already done that, but in ns longhorn 
storage-network: longhorn-system/{{ storage_macvlan_interface_name }}
👍 1
w
have tried exec'ing into a pod to check addresses, routes and connectivity?
I spent a bunch of time debugging before I did that 🤦
s
Which CNI do you use?
pod to pods ok pods to other nodes ok pod to underlying node nok
w
ah, this the macvlan thing
you need a macvlan interface on the host
s
but the macvlan interface should be attached to a network namespace
w
not sure what you mean
I'll find the commands I used
s
🙏
w
that's a snippet from the 
/etc/network/interfaces
file
it creates a new interface called 
shim-storage0
and links it to 
storage0
as a macvlan bridge
then puts the host's IP address for the storage network on the new interface
this way the host and the containers are all on the same bridge
s
hum I’ve tried to do this with network manager, I’ve surely miss up something
w
oh, yeah I would disable network manager and do it manually
s
thank you very much ! I’ll try it
👍 1
w
I disabled connmand on k3os to get this to work
btw, I renamed 
eth1
to 
storage0
, you don't have to do that
s
don’t really fan of network manager but they have a pretty good ansible module
w
you could potentially disable Network Mangler just for the storage network interface
s
thanx! it was succesfull with a mix of nmcli and ip, now I’ll automate that 🤯🔫
w
woohoo \0/
it was cool to watch the graphs when I pushed 15GB to an r=3 volume. only one node saw the traffic on the main NIC, all nodes saw traffic on the storage NIC
174 Views
Open in Slack
PreviousNext
Powered by