My bad, UI has scroll down, which is not that visible. Add disk is there!

I'm reading the longhorn documentation and looking over the CSI Persistent Volume example. I see the spec.csi.volumeHandle parameter as pointing to an already created longhorn volume and trying to understand the usage of the csi section. What I've been trying to find is a reference of the options that are possible under the spec.csi section - I figured there must be a reference somewhere but can't find one, just examples but not a formal spec. I've even tried reading the Go code and haven't found where that section is parsed. Can someone steer me in the direction of a definitive reference for this section?

I just found this discussion which is I think what I've been struggling to understand: https://github.com/longhorn/longhorn/discussions/2190

Is there anything in longhorn that runs in a strict 30s loop? Getting these strange errors across a cluster running longhorn volumes, every 30 seconds on each node, but I have no workloads that fit such a pattern, and these errors don’t surface to any running workloads as far as I can tell (I have a rancher prometheus volume that’s stuck in a perpetual attach/detach loop, but that’s much more frequent.. it keeps looping even when the statefulset is scaled to 0):

[Thu Mar  9 09:13:02 2023] Buffer I/O error on dev dm-2, logical block 2441200, async page read
[Thu Mar  9 09:13:32 2023] buffer_io_error: 23 callbacks suppressed
[Thu Mar  9 09:13:32 2023] Buffer I/O error on dev dm-0, logical block 25837552, async page read

I have seen it several times that one of my nodes got stuck with a bazillion of messages like

Mar 12 00:00:02 srvl062 kernel: [713683.338101] nfs: server 10.43.170.7 not responding, timed out
Mar 12 00:00:03 srvl062 kernel: [713683.914078] nfs: server 10.43.187.251 not responding, timed out
Mar 12 00:00:03 srvl062 kernel: [713684.170058] nfs: server 10.43.28.105 not responding, timed out
Mar 12 00:00:05 srvl062 kernel: [713685.453982] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:05 srvl062 kernel: [713685.610363] nfs: server 10.43.133.91 not responding, timed out
Mar 12 00:00:05 srvl062 kernel: [713686.217960] nfs: server 10.43.28.71 not responding, timed out
Mar 12 00:00:06 srvl062 kernel: [713687.245956] nfs: server 10.43.43.8 not responding, timed out
Mar 12 00:00:08 srvl062 kernel: [713689.289855] nfs: server 10.43.131.82 not responding, timed out
Mar 12 00:00:10 srvl062 kernel: [713690.501785] nfs: server 10.43.170.7 not responding, timed out
Mar 12 00:00:10 srvl062 kernel: [713690.569795] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:13 srvl062 kernel: [713693.385742] nfs: server 10.43.187.251 not responding, timed out
Mar 12 00:00:13 srvl062 kernel: [713694.185659] nfs: server 10.43.43.8 not responding, timed out
Mar 12 00:00:14 srvl062 kernel: [713695.013664] nfs: server 10.43.28.105 not responding, timed out
Mar 12 00:00:15 srvl062 kernel: [713695.693620] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:16 srvl062 kernel: [713696.521595] nfs: server 10.43.133.91 not responding, timed out
Mar 12 00:00:20 srvl062 kernel: [713700.557466] nfs: server 10.43.170.165 not responding, timed out
Mar 12 00:00:20 srvl062 kernel: [713700.617405] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:20 srvl062 kernel: [713700.813461] nfs: server 10.43.131.82 not responding, timed out
Mar 12 00:00:21 srvl062 kernel: [713701.641479] nfs: server 10.43.187.251 not responding, timed out
Mar 12 00:00:22 srvl062 kernel: [713703.241337] nfs: server 10.43.70.182 not responding, timed out
Mar 12 00:00:24 srvl062 kernel: [713704.717343] nfs: server 10.43.170.7 not responding, timed out
Mar 12 00:00:25 srvl062 kernel: [713705.485281] nfs: server 10.43.202.97 not responding, timed out

What is the recommended procedure to get out of this?

set the channel topic: The latest Longhorn release is v1.4.1. Community event at: https://community.cncf.io/cncf-longhorn-project/

@aloof-hair-13897 do you happen to know why longhorn 1.4.0 never made it to rancher marketplace? If not, that would be my #1 topic to mention on the community call. We're already on 1.4.1 and my expectations of seeing it in Rancher are quite low now... 😂 I was thinking of installing the Helm version but it's too late now...

Is there some way to disable NFS support in Longhorn to improve stability?

Hey, I don’t suppose there is any unsupported way to use an NFS v3 server in Longhorn. Damn Mac OS only has v3 server (along with SMB and AFS) and thats where my big disks are shared. This is only for home/tinkering use!

I wanted to flag that I think this is not a good design.

failed to set the setting priority-class with invalid value rancher-critical: cannot modify priority class setting before all volumes are detached

The change should be put in place and applied when possible, without causing downtime. There is no easy way to detach all volumes without scaling all workload to 0, which is painful. Maybe a kill/maintenance switch in longhorn can be implemented to facilitate such changes in an easier manner.

how would you suggest to safely detach volumes out from under a running workload?

@numerous-lighter-90852 has left the channel

So I, silly me, did not follow https://longhorn.io/docs/1.4.1/volumes-and-nodes/maintenance/#updating-the-node-os-or-container-runtime when I removed a node by first draining it … from a k3s v1.25.5 cluster. I can create a simple pod that consumes a pvc and that gets up and running and I can exec into the pod and write data to a mount on the created pvc. But, two of the volumes on the cluster - NOT RWX - cannot get attached. I’ve hard deleted them but they end up getting deadline exceeded errors after some time and thereby fail to attach. Tailing all pods in the longhorn-system namespace, with the tail krew plugin, I see that the removed node is mentioned with a log line stating that the engine image is not ready on the removed worker. And of course not - because the worker is gone. How can the above situation be remediated? This issue > https://github.com/longhorn/longhorn/issues/5408 is somewhat similar. Thank you very much

Looking at the longhorn volume crd’s representing the pvc’s that cannot be attached and the Old removed worker node is listed in the pendingNodeID field

Hmm a restore of a volume via the backup panel in the longhorn ui made things clear up …. Really need to ensure I follow the steps needed when removing a node that’s a “longhorn” node in the cluster ….

Still interested in knowing if there’s a better than restoring in the situation that happened here ( besides following the guide ) - as my impression is that these attach issues can come from a plethora of different “places”

I am always seeing this issue with longhorn admission webhook. The current version I am on is 1.3.2 Does anyone faced this issue before or got any ideas why this might be happening. Thanks.

Interesting longhorn replica eviction behavior I bumped into. So: • disabled scheduling on all disks on a longhorn node • Set eviction to true on the disks on the same node As there’s no other node to actually “take on” the evicted replicas nothing happens. I enable disk scheduling again. However, without setting eviction requested to false on all the disks on the node. The replicas on the node gets doubled. I know I may be fooling around a bit here. But, still - isn’t this unintended behavior? Thank you

Another question. Isn’t the effect of disabling scheduling at the node level. The same as compared to disabling scheduling on all disks on the same node?

Feel VERY free to chime in on this discussion > https://github.com/longhorn/longhorn/discussions/5593 < 😄 thank you!

Hi, is it possible to assign a persistent storage volume to an already running ‘app’, using Rancher/k3s and Longhorn?

hi All, i have 10 node cluster and by using longhorn created the dynamic storage ….. to check the resilience bit i tried to down two nodes and checked the replica is creating in another node…. but still it is not creating replicas in other available nodes ?

Hello, one of the volumes keeps "attaching", how do I cancel and attach it to another node?

Hello, longhorn has been giving me headache lately (I'll ask other questions later), but the most recent issue ends up making me being stuck so far. I am deploying jupyterhub, it's been working fine but not today. I get this error for instance when one of the pods is trying to use the PV. I see the PV/PVC created properly. The volume is reported healthy in longhorn UI.

MountVolume.MountDevice failed for volume "pvc-2e1be23b-4021-45d2-878b-d37cd71679dc" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc" failed: type:("ext4") target:("/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/c072c61d6cbcb55eac5a9120cce1da05cba3ab07bb0b540278a9cfcc67b87887/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)...

it's running on a microk8s cluster.

Hi, when removing node from a cluster, I want to make a eviction request to Longhorn volume on that node first. Is there any API I can call to do it programmatically?

Will my replicas be sensibly, automatically distributed? How can I verify this? I am testing my understanding of replicas. Suppose I have two nodes each having a single fast disk and two slow disks. I have defined a storage class for each: fast-ephemeral, slow-durable. To suit this use case, I set two replicas for the fast-ephemeral class, and four for the slow-durable class. Is it correct to expect LongHorn to place two slow-durable replicas on each node on different disks automatically, and one fast-ephemeral replica on each node? Thank you your time.

Hello all, Michael from GeoInsight here, trying to test out k3s, rancher and longhorn. Following the tutorial `

Following the tutorial

kubectl get pods --namespace longhorn-system --watch

I get:

NAME                                           READY   STATUS             RESTARTS         AGE
longhorn-admission-webhook-56d6cdc68f-2c8f4    0/1     Init:0/1           0                4h50m
longhorn-admission-webhook-56d6cdc68f-5mf9n    0/1     Init:0/1           0                4h50m
longhorn-admission-webhook-668d9787dc-r8nnd    0/1     Init:0/1           0                4h19m
longhorn-conversion-webhook-55cf57895c-5dtc7   1/1     Running            0                4h50m
longhorn-conversion-webhook-55cf57895c-cn68m   1/1     Running            0                4h50m
longhorn-driver-deployer-77c46b4f54-gxnsx      0/1     Init:0/1           0                4h50m
longhorn-manager-bn4pr                         0/1     Init:0/1           0                4h50m
longhorn-manager-v9jm7                         0/1     Init:0/1           0                4h50m
longhorn-recovery-backend-ccfc78cb6-pbxfj      1/1     Running            0                4h50m
longhorn-recovery-backend-ccfc78cb6-pnxsz      1/1     Running            0                4h50m
longhorn-ui-b8c58884-fr45d                     0/1     CrashLoopBackOff   57 (4m27s ago)   4h50m
longhorn-ui-b8c58884-wq2gm                     1/1     Running            0                4h50m

So a lot of pods are in

Init:0/1

and stuck there. I narrowed it down to this error in the

longhorn-admission-webhook

which depends on

longhorn-conversion-webhook

it seem:

Defaulted container "longhorn-admission-webhook" out of: longhorn-admission-webhook, wait-longhorn-conversion-webhook (init)
Error from server (BadRequest): container "longhorn-admission-webhook" in pod "longhorn-admission-webhook-56d6cdc68f-5mf9n" is waiting to start: PodInitializing

But the log for

longhorn-conversion-webhook

seems fine. or not?

$ kubectl --namespace longhorn-system logs longhorn-conversion-webhook-55cf57895c-5dtc7
time="2023-03-27T06:58:39Z" level=info msg="Starting longhorn conversion webhook server"
W0327 06:58:39.336261       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2023-03-27T06:58:39Z" level=warning msg="Failed to init Kubernetes secret: secrets \"longhorn-webhook-tls\" not found"
time="2023-03-27T06:58:40Z" level=info msg="Listening on :9443"
time="2023-03-27T06:58:40Z" level=info msg="certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2023-03-27 06:58:36 +0000 UTC notAfter=2024-03-26 06:58:40 +0000 UTC"
time="2023-03-27T06:58:40Z" level=info msg="Updating TLS secret for longhorn-webhook-tls (count: 1): map[<http://listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc|listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc> <http://listener.cattle.io/fingerprint:SHA1=8AE712382F7744254DB21B7BE49563465445AA91]|listener.cattle.io/fingerprint:SHA1=8AE712382F7744254DB21B7BE49563465445AA91]>"
time="2023-03-27T06:58:41Z" level=info msg="Active TLS secret longhorn-webhook-tls (ver=10272145) (count 1): map[<http://listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc|listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc> <http://listener.cattle.io/fingerprint:SHA1=8AE712382F7744254DB21B7BE49563465445AA91]|listener.cattle.io/fingerprint:SHA1=8AE712382F7744254DB21B7BE49563465445AA91]>"
time="2023-03-27T06:58:42Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-03-27T06:58:42Z" level=info msg="Building conversion rules..."
time="2023-03-27T06:58:42Z" level=info msg="Starting <http://apiextensions.k8s.io/v1|apiextensions.k8s.io/v1>, Kind=CustomResourceDefinition controller"
time="2023-03-27T06:58:42Z" level=info msg="Starting <http://apiregistration.k8s.io/v1|apiregistration.k8s.io/v1>, Kind=APIService controller"
time="2023-03-27T06:58:42Z" level=info msg="Updating TLS secret for longhorn-webhook-tls (count: 1): map[<http://listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc|listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc> <http://listener.cattle.io/fingerprint:SHA1=8AE712382F7744254DB21B7BE49563465445AA91]|listener.cattle.io/fingerprint:SHA1=8AE712382F7744254DB21B7BE49563465445AA91]>"
time="2023-03-27T06:58:42Z" level=info msg="Update CRD for <http://nodes.longhorn.io|nodes.longhorn.io>"
time="2023-03-27T06:58:43Z" level=info msg="Update CRD for <http://volumes.longhorn.io|volumes.longhorn.io>"

I've had someone suggest that Longhorn has S3 compatibility. I've been trying to find evidence for this, but am failing. Seems like it is more about managing PVCs. Is that correct?