We have requirement, where we have provision k3s cluster with x2 master nodes and x3 worker nodes for storage cluster with longhorn and we need this storage cluster to be used / provision in another kubernetes cluster ( workload cluster) PV / PVC claims. The idea is to provide PV/ PVC claims for software stack such as harbor , ELK ( elastic search), custom application workload that need storage for batch processing;
Does it make sense to;
• Option 1: Connect k3s - storage cluster to workload cluster (2nd k8s cluster) via means of NFS service from ingress ( or any other means) for production work ( i would like to keep longhorn capabilities on top and expose the storage through nfs-server. I saw somewhere that longhorn itself has nfs-server pod as well with longhorn stack)
• Option 2: Create workload pods in same k3s - storage cluster having seperate nodes ( by grouping / labeling nodes) to run application workload
• Option 3: Any other options??
Any insights/ feedback in to identify the correct approach is highly appreciated.