https://rancher.com/ logo
Join Slack
Powered by
Hello folks, I followed <https://github.com/harves...
# harvester
g
Hello folks, I followed https://github.com/harvester/harvester/issues/6975 to create a storageClass using SAN LUNs on Harvester v1.4.2; I enabled multipath, created a customization file in /oem to add entries in /etc/fstab at boot time, nodes have the SAN disks mounted after reboot, then used the Longhorn UI to add the disks into the nodes, assigned node and disks tags, created a storageClass with nodeselector and diskselector. All seems fine but when I create a PVC from that storageClass I get "No available disk candidates to create a new replica of size ... " and "Cannot find a reusable failed replicas for volume pvc-xxx" errors on the longhorn-manager pods. Anyone can point me where I can get some more logs/troubleshooting info? Thanks in advance
b
You have good output from 
multipath -ll
?
g
yes, I have 4 paths per disk as expected, I did a dd test and write speed is also good
b
I'm a little confused as it looks like the harvester issue you posted is for adding the SAN LUNs to the system for use with longhorn and not a seperate Storage class.
Can you dump your storage class yaml thats failing?
g
I used a new storageClass because I only map LUN's to two out of 3 nodes (and I configured 2 replicas for this SC); yes I can paste the SC yaml, give me a sec; I can also take a support bundle
b
The bundle might be good for some Suse folks, but I won't mess with it. (just an Fyi)
So here's the thing: If you're adding the path that the LUNs are mapped to as per that last comment in the github issue that's listed as a workaround, the SAN LUNs are abstracted from the k8s cluster. Longhorn (the storage provider) just sees it as another "Disk" on the system. It doesn't care how it's connected it's just raw space. Now... you can add labels to that storage "device" in longhorn to set expectations because of lag/speed. (Maybe your san is too slow for etcd or something) Just like you could for spinning rust vs ssd vs nvme, but by default longhorn is just gonna plop things wherever. It doesn't even matter that only two of the nodes have the SAN attached. It'll make sure there's three copies of the volume somewhere.
g
multipath_out.txt
multipath_out.txt
b
That actually might be a really good thing. If your SAN takes a dump, you'd potentially still have a copy on a hyperconverged local host somewhere and you'd be ok.
t
I would test with 1.5.0 and add a CSI to talk directly to the FC array. Avoid longhorn for this.
b
That's pretty good advice too
Though it means you won't be able to take backups.
t
You can! on the array.
b
Via the array, but not via Harvester
t
agreed.
g
I could probably try that, not sure how can I add the CSI but I'll do some reading
b
diskSelector: vm
means you shouldn't need to have the node selector
🙌 1
if the longhorn disks have that label on them
g
I remember there was some IBM block operator for FlashSystems, not sure about these old Storwize arrays I have in my lab
t
take a look at

https://youtu.be/5VB8fHJuAQQ▾

b
I would change it to something more obvious like 
sanlun
👍 1
t
you will need 1.5.0-rc4
✅ 1
g
@bland-article-62755 I re-created the storageClass with only the diskSelector tag (without nodeSelector) and the volumes are created successfully now, thanks alot for the suggestion!
👍 1
b
🎉
g
I will probably test with the custom CSI as well on some VM's, just for fun
b
having backups through Harvester is worth dealing with the overhead imho. Besides you already did all the hardest parts with getting it into the node.
You can always have both.
But backups give you a way to migrate VMs to other clusters.
g
yes, longhorn managed backups are nice to have
although at some point (for this lab) re-mapping the luns to a different host or snapshoting at the stoarge level could also work
4 Views
Open in Slack
PreviousNext
Powered by