@billowy-painting-56466 I installed Multus and 3 network adapters on 3 nodes of 14 nodes cluster. In the documentation https://longhorn.io/docs/1.7.1/references/settings/#storage-network There are two unclear mentions: 1. set

storageNetwork: kube-system/demo-192-168-0-0

I think since I did my

NetworkAttachementDefinition

in

longhorn-system

namespace and named it

macvlan-longhorn

that my setting in the helm chart should be:

storageNetwork: longhorn-system/macvlan-longhorn

In the documentation it's also mention:

Prerequisite
The Multus NetworkAttachmentDefinition network for the storage network setting must be reachable in pods across different cluster nodes.

What I understand is that ALL PODS across different cluster nodes. Question: I have only 3 nodes that have the PRIVATE VLAN Network. Would this work or all nodes needs to have a NIC on the VLAN ? Then you have this Warning:

Warning: This setting should change after all Longhorn volumes are detached because some pods that run Longhorn system components are recreated to apply the setting. When all volumes are detached, Longhorn attempts to restart all Instance Manager and Backing Image Manager pods immediately. When volumes are in use, Longhorn components are not restarted, and you need to reconfigure the settings after detaching the remaining volumes; otherwise, you can wait for the setting change to be reconciled in an hour.

So I understand that if I wait 1 hour the

Instance Manager

will have the annotation added and get a new IP from the VLAN but what about the ones that don't have a NIC to the VLAN ?

As soon as I bind an IP address on one of the nodes secondary NIC connected to the VLAN it takes few minutes to 1 hour and the routing is not working anymore in the cluster not only for longhorn. I need to have a more detailed documentation to make sure the steps are not disrupting the cluster. I also need to know how to tell Longhorn that it's only on certain nodes that this network can be used.