https://rancher.com/ logo
Title
n

numerous-kilobyte-30360

07/01/2022, 9:39 PM
Hi! I'm new to k3s, coming from Docker Swarm in a small home lab environment. In my k3s deployment, all (3) nodes have shared storage already, located under
/mnt/storage
. For this reason, I don't want to use Longhorn. I'd like to use my existing replicated storage. k3s seems to default to
/var/lib/rancher/k3s/storage
with the local storage provider. I see an option for use during initial k3s setup,
--default-local-storage-path
per the documentation at https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#storage-class The setting appears to be located in
/var/lib/rancher/k3s/server/manifests/local-storage.yaml
on each node, but it also appears from reading around on forums that this gets overwritten and should be changed with tools, rather than directly. • Is it possible to change the
--default-local-storage-path
parameter without reinstalling k3s? • Is this even the correct parameter to change? Thanks!
c

creamy-pencil-82913

07/01/2022, 9:42 PM
the local-path-provisioners is for just that, local paths. It doesn’t expect anything in there to be replicated across nodes, even if it is it won’t be aware of that.
If you have an existing NFS server that you’re using, setting up the external NFS provisioner is probably your best bet
n

numerous-kilobyte-30360

07/01/2022, 9:43 PM
I do not have nfs. It's ceph, provisioned by proxmox, with block storage made available inside each node.
c

creamy-pencil-82913

07/01/2022, 9:43 PM
ah. Then you might be able to use the Ceph CSI driver
There’s no real way to just tell kubernetes “this is the same shared filesystem on all the nodes”. You really need to go through the volume abstraction layer, and the best way to do that is to find a CSI driver that works with your storage backend.
n

numerous-kilobyte-30360

07/01/2022, 9:46 PM
I see. Alright, I'll look into doing it that way. Thanks!
Looks like the status is not in a great place for ceph. Alpha, with no targets listed for beta or GA - https://kubernetes.io/blog/2021/12/10/storage-in-tree-to-csi-migration-status-update/
c

creamy-pencil-82913

07/01/2022, 9:57 PM
That’s for migration, if you’d been using the in-tree volume driver and are trying to migrate to the CSI driver. The standalone CSI driver is quite mature. https://github.com/ceph/ceph-csi
n

numerous-kilobyte-30360

07/01/2022, 9:59 PM
This looks good. Thank you!