Yeah, but doc says host fs should be ext4

the devices used by longhorn should be

Hi! If I mount a volume using RWX in Longhorn, share-manager builds the NFS share to the following path:

/dev/longhorn

Is there a way to change this path? I didn't find any arg in Helm chart vaules file (https://github.com/longhorn/charts/blob/master/charts/longhorn/values.yaml).

Anyone here with some input on what’s considered best practice in regards to the combo of Longhorn<>Velero .. in regards to cleaning up snapshots. • Our Velero is set to retain 5 backups … and CSI Snapshots is enabled ◦ Velero triggers Longhorn via the

VolumeSnapshotClass

• Longhorn backs up and is configured to integrate with MinIO to store the data … We’re seeing a lot of storage usage and after having read > https://longhorn.io/docs/1.3.1/volumes-and-nodes/volume-size/ I’m thinking about if it’ll be beneficial to configure a

snapshot recurring job

with a

retention of 1

in order to get these snapshots taking up storage space to reclaim disk space. As my thinking/understanding is that the data is stored on MinIO anyways and

Velero

tracks the data on MinIO and on restore will trigger

Longhorn

- again over

CSI

VolumeSnapshot

for restore … and the data will be fetched from

MinIO

. So the data in all these

snapshots

that one can see on the sub-volume page for a specific

Volume

can be reclaimed and we still have backup data in the case we need it for restore. Is this correctly understood? And thank you very much.

Hi All, I would very much like some help if possible. So I have got a small cluster setup using Rancher RKE2, and wanted some persistent storage for an app I installed longhorn, and the iscsi tools on each of the nodes and have them running, and in longhorn now have a healthy iSCSI volume, and using longhorn created the Persistent Volume and Persistent Volume Claim as my app needed I then attempted to deploy the app to be met by this errors:

Unable to attach or mount volumes: unmounted volumes=[<app>-pvc]

AttachVolume.Attach failed for volume "<app>" : rpc error: code = InvalidArgument desc = volume <app> invalid frontend type iscsi

I found this page that may have some more information: https://docs.ranchermanager.rancher.io/v2.5/how-to-guides/advanced-user-guides/manage-cluste[…]orage/manage-persistent-storage/install-iscsi-volumes and the next instruction 'After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below' I don't know which YAML it is referring to, any help here would be great, thanks in advance

With the latest version ( v1.3.2 ) using helm to deploy. On a k3s v1.24.4+k3s version cluster - helm blows up with the version of the kubernetes cluster being incompatible and that v1.18 <> v1.25 is needed … I also tried using the specific v1.3.x branch on the longhorn helm chart repo. And further tried deleting my helm cache. —- Any ideas? Thank you

why I have 3x dir

8.0G	pvc-48586aa6-cc79-40e0-94b1-bf20e48ffa72-01eb5a76
28G	    pvc-48586aa6-cc79-40e0-94b1-bf20e48ffa72-6f9f75b9
8.5G	pvc-48586aa6-cc79-40e0-94b1-bf20e48ffa72-ca55e8c5

on single node to one PVC?

Hey folks, we have been using rook/ceph for a couple of years and we are now experimenting with longhorn for its multi-cluster volume replication feature. we were expecting longhorn to have some sort of disk autodiscovery feature that can discover and format raw disks attached to the nodes just like rook/ceph does. but it doesnt seem to be the case in default helm chart installation. instead, we have to mount/format all the disks manually and then add them to longhorn in the longhorn UI. my question is: is there any existing/mature solutions that can cover this disk autodiscovery/formatting experience?

Hey, im having trouble reinstalling longhorn. I have an older version that I didn't follow through with proper uninstall, which is my fault. So I tried running all the "uninstall.yaml" versions based on the major releases in the github branches. Everything after 0.4 shows this:

E1018 11:12:14.544813       1 reflector.go:126] <http://github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117|github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117>: Failed to list *v1beta1.Setting: the server could not find the requested resource (get <http://settings.longhorn.io|settings.longhorn.io>)
E1018 11:12:14.745037       1 reflector.go:126] <http://github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117|github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117>: Failed to list *v1beta1.Engine: the server could not find the requested resource (get <http://engines.longhorn.io|engines.longhorn.io>)
E1018 11:12:14.944640       1 reflector.go:126] <http://github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117|github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117>: Failed to list *v1beta1.Replica: the server could not find the requested resource (get <http://replicas.longhorn.io|replicas.longhorn.io>)
E1018 11:12:15.144968       1 reflector.go:126] <http://github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117|github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117>: Failed to list *v1beta1.InstanceManager: the server could not find the requested resource (get <http://instancemanagers.longhorn.io|instancemanagers.longhorn.io>)
E1018 11:12:15.367282       1 reflector.go:126] <http://github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117|github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117>: Failed to list *v1beta1.EngineImage: conversion webhook for <http://longhorn.io/v1beta2|longhorn.io/v1beta2>, Kind=EngineImage failed: Post "<https://longhorn-conversion-webhook.longhorn-system.svc:9443/v1/webhook/conversion?timeout=30s>": service "longhorn-conversion-webhook" not found

So some webhook was deleted before it should have been? Further evident when I try to remove the finalizers from some objects manually,

   ~  kubectl edit <http://nodes.longhorn.io|nodes.longhorn.io> -n longhorn-system                                                                                                                                   ✔  red ⎈ 
error: <http://nodes.longhorn.io|nodes.longhorn.io> "mcsh1" could not be patched: Internal error occurred: failed calling webhook "<http://validator.longhorn.io|validator.longhorn.io>": Post "<https://longhorn-admission-webhook.longhorn-system-2.svc:9443/v1/webhook/validaton?timeout=10s>": service "longhorn-admission-webhook" not found
error: <http://nodes.longhorn.io|nodes.longhorn.io> "mcsh2" could not be patched: Internal error occurred: failed calling webhook "<http://validator.longhorn.io|validator.longhorn.io>": Post "<https://longhorn-admission-webhook.longhorn-system-2.svc:9443/v1/webhook/validaton?timeout=10s>": service "longhorn-admission-webhook" not found
error: <http://nodes.longhorn.io|nodes.longhorn.io> "pfdc-store-2" could not be patched: Internal error occurred: failed calling webhook "<http://validator.longhorn.io|validator.longhorn.io>": Post "<https://longhorn-admission-webhook.longhorn-system-2.svc:9443/v1/webhook/validaton?timeout=10s>": service "longhorn-admission-webhook" not found
error: <http://nodes.longhorn.io|nodes.longhorn.io> "ryzen1" could not be patched: Internal error occurred: failed calling webhook "<http://validator.longhorn.io|validator.longhorn.io>": Post "<https://longhorn-admission-webhook.longhorn-system-2.svc:9443/v1/webhook/validaton?timeout=10s>": service "longhorn-admission-webhook" not found

All and all, I have these resources left based on the status of this NS

- lastTransitionTime: "2021-11-04T05:49:28Z"
    message: 'Discovery failed for some groups, 3 failing: unable to retrieve the
      complete list of server APIs: <http://subresources.kubevirt.io/v1|subresources.kubevirt.io/v1>: the server is currently
      unable to handle the request, <http://upload.cdi.kubevirt.io/v1alpha1|upload.cdi.kubevirt.io/v1alpha1>: the server is
      currently unable to handle the request, <http://upload.cdi.kubevirt.io/v1beta1|upload.cdi.kubevirt.io/v1beta1>: the
      server is currently unable to handle the request'

How do I completely remove longhorn from here so I can reinstall it on my cluster?

I have longhorn running as my default storage class for multiple clusters. The original k3s StorageClass of

local-path

was patched to unset it as a "default", but I have noticed that over time the

local-path

comes back as a default storage class. I've unset it several times now but I'd like it to stop reverting back to that state. Does anyone know how I can accomplish this or have you seen this yourself?

Hi! What user does Longhorn use when connecting to NFS backupTarget? I didn't find a mention in the documentation or chart/values.yaml. Did I miss something?

it is possible read contnet of pvc direclty on host instead of Pod ? I' mean no the .img file but the content of pod using storageclass

Is it possible to add a toleration without having to detach all volumes? Or a process to do it without having to take everything down to detach?

I have three nodes in my Harvester cluster which all seem to be working correctly, but Longhorn has several degraded volumes (like the one pictured) which is will not rebuild onto one of the nodes. Does anyone have any ideas why that might be, and how I can fix it? I've previously asked on the #harvester channel but the trail went cold.

I'm not sure what's gone wrong but I have a volume that can't attach anywhere, my logs are showing the following despite the UI saying it's detached:

E1024 14:49:44.621450       1 volume_controller.go:216] fail to sync longhorn-system/pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af: fail to reconcile volume state for pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af: volume pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af has already attached to node kube-w-04, but asked to attach to node kube-w-03
time="2022-10-24T14:49:44Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af out of the queue" controller=longhorn-volume error="fail to sync longhorn-system/pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af: fail to reconcile volume state for pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af: volume pvc-336ed0a5-1889-4e0a-be69-e9cd6602b4af has already attached to node kube-w-04, but asked to attach to node kube-w-03" node=kube-w-03

How can I tell it that it's not attached to w-04?

I’m running k3s v1.24.7+k3s1 and longhorn v1.3.1. Are these two compatible with Ubuntu 22.04 ( jammy )? It seems to me that longhorn did not bootstrap the nodes to be longhorn nodes. No nodes in the longhorn ui. Just a blank ui. Thank you very much.

Hello,

I have a question about longhorn backup, the backup configuration in our system is : S3://clustername-longhorn/us-east1. My understanding is that it is using an test local S3 (Minio) storage. But I can't figure out where the S3 storage is. Can anyone help pointing me to the right direction where I can find the configuration for S3 Server is. I searched many places in the internet, but can't find it. Your help would be appreciated. Martin

What does it mean when a volume has two instance manager's listed, and errors saying there are two instance managers, and the volume is healthy and not ready, and attached when the Harvester VM is not running. It seems to be a Schrodinger's volume... I think it needs to be unattached with no instance managers, so that it can be attached again when Harvester commands it to be.

Has anyone faced an issue where even setting the taints, tolerations, and node selector, the longhorn workloads still go to the wrong nodes? For instance, we have 2 nodes dedicated to database related workloads, including longhorn, and they are being tainted and labelled correctly. I have made the needed changes on Longhorn-ui, Longhorn manager, and the driver deployer, but still the CSI components for example are on the wrong nodes (non tainted). Not really sure what is going on, I have been looking into this for the past 2 days. I have been trying to get a hold of someone from longhorn team but had no luck so far. I saw some issues opened about that, but all of them were resolved with the taints and tolerations setting.

Is there any roadmap where I can see when longhorn will support k8s 1.25, I need to upgrade my cluster because I need a feature from 1.25 but when I tried to create a test environment I saw that longhorn supports up to k8s 1.24

Trying to use another disk on my harvester node but can't seem to find the drop down in the longhorn UI. harvester 1.1/longhorn 1.3.2 from the longhorn docs, "To add a new disk for a node, head to the

Node

tab, select one of the nodes, and select

Edit Disks

in the dropdown menu." any ideas... command line or gui is fine.

i was wondering whether there was any way to label a volume to add a recurring backup job? i am sure that i read something on that but i cannot see the reference. many thanks.

more longhorn specific questions... I have added a few nodes to my pool with more CPU and memory and am draining and cordoning the older nodes before I decommission them. How would I safely remove this instance manager pods from these original nodes. force eviction results in: error when evicting pods/"instance-manager-r-f71e78fc" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

it was just a temporary lab cluster so I deleted it in the end and recreated the pool with more CPU and memory nodes. Would be great to understand in the future how to remove those instance-manager pods

I messed up with uninstalling and now the namespace is stuck on "Terminating", I don't seem to have any stale resources left besides the namespace, which prevents me from installing it again 😕 What I initially did was attempting to uninstall just by deleting the namespace, unfortunately I only read the documentation about uninstalling after that, and it doesn't seem to help at this point.

$ kubectl get all -n longhorn-system
No resources found in longhorn-system namespace.

$ kubectl get namespace longhorn-system
NAME              STATUS        AGE
longhorn-system   Terminating   104m

should I just rebuild the cluster? 😢

Hello Everyone, We installed longhorn for storage class using kubectl and everything was working well when we used the root disk for storing all the data. Now I created an addtinal disk and attached it to the VM by creating a lvm. This was done specifically for /var/lib/longhorn folder as this contains the PVs.

-> df -h
Filesystem                 Size  Used Avail Use% Mounted on
/dev/sda3                  6.0G  1.5G  4.5G  24% /
devtmpfs                   7.9G     0  7.9G   0% /dev
/dev/sda2                   93M   32M   55M  37% /media/hdCnf
/dev/sda8                   43G  6.7G   34G  17% /media/hdVar
/dev/sda5                   93M   28K   86M   1% /media/hdUser
/dev/loop0                 2.0G   13M  1.9G   1% /media/loopfs
cgroup                     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda7                  100M   69M   31M  69% /boot
/dev/sda1                  200M   69M  131M  35% /run/media/sda1
/dev/mapper/vg01-longhorn  177G   29M  168G   1% /var/lib/longhorn

But now the PV are not getting attached to the pods and keeps giving error message failed to attach the volume. Any idea why this might happening.

Hi Everyone, I'm working on longhorn and it is working well and maintainability from ui is very easy by defining tags, adding disks, backups and snapshots. But i did not find the anything for programatic use cases like maintaining longhorn features like adding disk and tags to differentiate the hdd and ssd pool. please help me to point out any CLI scripts/apis where i can reference to use it in my environment.

Hi all, as is seemingly standard when joining channels like this... I think i've got an issue but it could also be RHEL and its horrible tx-checksum-ip-generic issue manifesting itself in a different way:

failed calling webhook \"<http://validator.longhorn.io|validator.longhorn.io>\": failed to call webhook: Post \"<https://longhorn-admission-webhook.longhorn-system.svc:9443/v1/webhook/validaton?timeout=10s>

Any pointers welcome - I'm about to fire up a dnsutils pod to look at the DNS now.

Longhorn in my Harvester has been working perfectly for a while. This morning I noticed that one of my VMs has stopped. I looked in the Longhorn UI and I see that the root disk volume is not happy. Whenever that volume is mounted it goes into an "Attaching" -> "Detaching" loop. How can I work out what happened? And how I can get that that partition back into a healthy mode?