I'm having some trouble installing NueVector due to RWX volume requirements, however - I see RWX should be available now - accoring to - https://github.com/harvester/harvester/issues/1992 I'm running Harvester 1.4.1, Rancher 2.11.2, cluster has harvester-csi-driver:0.1.2300. Loosely following the issue above, in harvester I have a storage class called "ssd-rwx" as follows -

yaml
kind: StorageClass
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
metadata:
  name: ssd-rwx
provisioner: <http://driver.longhorn.io|driver.longhorn.io>
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  diskSelector: ssd
  staleReplicaTimeout: "2880"
  fromBackup: ""
  fsType: "ext4"
  nfsOptions: "vers=4.2,noresvport,softerr,timeo=600,retrans=5"

SSD just selectes the right disks, so I have a disk with the label "ssd" and it is available in the UI. In the target cluster I added the storage clas to match using the UI, YAML as follows -

yaml
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
kind: StorageClass
metadata:
  name: ssd-rwx
  fields:
    - ssd-rwx
    - <http://driver.harvesterhci.io|driver.harvesterhci.io>
    - Delete
    - Immediate
    - false
    - 54m
allowVolumeExpansion: false
parameters:
  hostStorageClass: ssd-rwx
provisioner: <http://driver.harvesterhci.io|driver.harvesterhci.io>
reclaimPolicy: Delete
volumeBindingMode: Immediate
__clone: true

I can create PVC's using this storage class as RWX - however when I try to mount them in a pod, I get the following error:

plaintext
MountVolume.MountDevice failed for volume "pvc-33157c69-b802-4bb5-9d13-ae00b0b4be76" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

The cluster nodes appear to have nfs-client installed - from one of the nodes -

bash
cluster-3-worker-bwjtp-7qxmw:/ # nfsstat -c
Client rpc stats:
calls      retrans    authrefrsh
1978       0          1978    

Client nfs v4:
null             read             write            commit           open             
9         0%     6         0%     0         0%     0         0%     4         0%     
open_conf        open_noat        open_dgrd        close            setattr          
0         0%     5         0%     0         0%     9         0%     4         0%     
fsinfo           renew            setclntid        confirm          lock             
28        1%     0         0%     0         0%     0         0%     0         0%     
lockt            locku            access           getattr          lookup           
0         0%     0         0%     28        1%     116       5%     30        1%     
lookup_root      remove           rename           link             symlink          
7         0%     2         0%     0         0%     0         0%     0         0%     
create           pathconf         statfs           readlink         readdir          
0         0%     21        1%     1561     78%     0         0%     20        1%     
server_caps      delegreturn      getacl           setacl           fs_locations     
49        2%     5         0%     0         0%     0         0%     0         0%     
rel_lkowner      secinfo          fsid_present     exchange_id      create_session   
0         0%     0         0%     0         0%     2         0%     1         0%     
destroy_session  sequence         get_lease_time   reclaim_comp     layoutget        
0         0%     63        3%     0         0%     1         0%     0         0%     
getdevinfo       layoutcommit     layoutreturn     secinfo_no       test_stateid     
0         0%     0         0%     0         0%     7         0%     0         0%     
free_stateid     getdevicelist    bind_conn_to_ses destroy_clientid seek             
0         0%     0         0%     0         0%     0         0%     0         0%     
allocate         deallocate       layoutstats      clone            
0         0%     0         0%     0         0%     0         0%

I see that on github you mention updating the CSI driver to the version on teh repo - "Upgrade csi-driver from 0.1.18 to 0.1.19 (with 0.2.0 image)" however the version provisoined by harvester seems to be newer than this so should version 0.1.2300 work? The issue is closed so just posting here, hoping to get some clarity on this as cant install NueVector without RWX support. Issue said milestone was 1.4.0 so have I missed something. @salmon-city-57654's verfy steps are the ones I was following - but I feel like I'm hitting a dead end... Are the versions I'm using supported, or have I missed something?

Have you seen the latest docs https://docs.harvesterhci.io/v1.4/rancher/csi-driver/#rwx-volumes-support You will need a storage network and to attach a NIC in the storage network to the VMs

I did see that, however the host cluster is running harvester 1.4.1, will take a quick look again tho. Also upgrading to 1.5 is our plan, but we'll go through the minot versions over the weekend

We have it "working" with 1.4.1 but there is a big issue you should be aware of https://github.com/harvester/harvester/issues/7796

We've a cluster installed on the harvester, in which we were trying to install nuevector, this requires RWX volumes - I hit this problem before and gave up in earlier harvester version and ended up running a dedicated NFS VM to service shared volumes, but now I need to sort it

Thanks, so you might hit the issue I linked. We also had issues with the NFS share that backs the RWX volume "locking up" from time to time and all writes failed until we restarted the guest cluster pods. Sadly overall not a reliable experience and we moved away from it.

Bit of a blocker for using nuevector then

ah yes, that was a pain

one for the weekend... will see if i can use existing nfs for nuevector install in the meantime

if you are shutting down make sure you have the storage network exclude IPs in the storage network settings in harvester, as that will also require a shutdown of all VMs

do i need to do that tho - as the vms use a completely different subnet

You need to attach a NIC from the storage network to the VMs hosting rke2 and then give that NIC an IP from the excluded range (so it doesn't conflict with anything harvester is doing)

ahh ic

where the eth1 IP we have set from the excluded range in the storage network

i get you, makes sense - on the other end we dedicated separate 10g ports for storage for each harvester node, so this needs to be routable to the pods then... think thats what your saying... 🙂 almost fudged this with our dedicated NFS by manually creating the PV...

We setup the guest rke2 cluster ourselves with ansible and our workload size is fairly static, all the IPs are set via ansible. Which isn't very dynamic but works for our use case. I don't think there is DHCP in the storage network to hook into but could be wrong.

ahh - i've been driving this from rancher itself, will give this a play when I get a chance to shut down fully

good luck!

Now i need to write all this up... thanks again neuvector is now running - all be it on poor mans nfs for now, I'll let you know how i get on with the RWX "proper" option later... though from the sounds of it I might be better off sticking with what we have!

Yeah, it might be more reliable with your one until that issue I linked is fixed

Might lay the ground work in the meantime - got to do 1.4.1->1.5.0 yet