rancher-users #longhorn-storage

busy-judge-4614

03/28/2023, 11:09 AM

Hi, is there a way to access the mounted Longhorn storage? Can I for instance add a directory?

busy-judge-4614

03/28/2023, 11:10 AM

Maybe this is it?

busy-judge-4614

03/28/2023, 11:10 AM

/dev/longhorn/media-store           49G  1.3G   48G   3% /var/lib/kubelet/pods/8230ce89-2723-432d-8a3c-35431ea7342c/volumes/kubernetes.io~csi/media-store/mount

bitter-tailor-6977

03/30/2023, 5:16 PM

Copy code

Warning  ExternalExpanding   2m36s                 volume_expand                        Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   Resizing            113s (x9 over 2m36s)  external-resizer <http://driver.longhorn.io|driver.longhorn.io>  External resizer is resizing volume pvc-705ba471-4f1a-4514-849a-e4d741964be6
  Warning  VolumeResizeFailed  113s (x9 over 2m36s)  external-resizer <http://driver.longhorn.io|driver.longhorn.io>  resize volume "pvc-705ba471-4f1a-4514-849a-e4d741964be6" by resizer "<http://driver.longhorn.io|driver.longhorn.io>" failed: rpc error: code = Internal desc = Post "<http://longhorn-backend:9500/v1/volumes/pvc-705ba471-4f1a-4514-849a-e4d741964be6?action=expand>": EOF

How to resolve this RWX volume expanding ?

steep-journalist-13696

03/31/2023, 7:06 PM

@steep-journalist-13696 has left the channel

late-needle-80860

04/06/2023, 7:37 PM

Seeking some advice on the best way of cleaning up backups in an automated way. We have quite the number of those to clean up. I’ve tried the Longhorn Python way and find it heavy. Reverse engineering the WebAPI that the Longhorn UI uses and then calling this with some Python code that piggy backs on the

requests

package gets me somewhere. But, for now I’m really interested in knowing what a solid approach actually is when one have to clean up a lot of backups? Where the case is that by mistake data was deleted on the S3 backend. Some the snapshot controller and the Longhorn CSI Snapshotter complains and tries to sync a lot of volumes it can’t find … Thank you

sparse-businessperson-74827

04/07/2023, 10:49 AM

Anyone can help me diagnose problem with longhorn 1.4.0. For some reason instance-manager pods shutdown on all nodes and then start multiple times a day at irregular intervals. I see shutdwon messages but cannot find the trigger. What is causing this?

Copy code

time="2023-04-07T10:37:26Z" level=warning msg="Received signal interrupt to shutdown"
[longhorn-instance-manager] time="2023-04-07T10:40:12Z" level=info msg="Shutdown all instance processes successfully"

agreeable-london-16334

04/09/2023, 8:58 AM

hi, im having some difficulties getting longhorn to use the fast ssds in my master nodes, they show up in the node list in longhorn ui and show the free space etc but when i try to create a volume it wont schedule on them. as soon as i make a disk in one of my worker nodes available the volume gets created on that, but my worker nodes only have tiny slow spinning disks. im guessing i am missing some config somewhere but not sure where (i dont see any pods failing to schedule due to taints etc, only due to persistent volume not being available)

✅ 1

bitter-tailor-6977

04/12/2023, 8:01 AM

hi all, https://github.com/longhorn/longhorn/blob/3f5e636bc3a53a1bddf1f4317185e11c8f208f17/deploy/longhorn.yaml#:~:text=engineManagerCPURequest i need the cpu value to be given predefined in charts…could any one help me to get the exact params and where to describe in the daemonset file… i dont want to edit from UI…i need to give predefined in manifest

hundreds-evening-84071

04/14/2023, 3:35 PM

Working on deploying Longhorn from Rancher (2.7.1) Apps. RKE2 v 1.23 I see this note: ----- Upgrading to Kubernetes v1.25+ Starting in Kubernetes v1.25, Pod Security Policies have been removed from the Kubernetes API. As a result, before upgrading to Kubernetes v1.25 (or on a fresh install in a Kubernetes v1.25+ cluster), users are expected to perform an in-place upgrade of this chart with

enablePSP

set to

false

if it has been previously set to

true

. ----- So, here my question... Where in the UI screens do I set

enablePSP

false

✅ 1

nutritious-oxygen-89191

04/18/2023, 7:08 AM

Hi all, I am getting

Copy code

Error from server: error dialing backend: x509: certificate is valid for 127.0.0.1, not <http://xxx.xxx.xxx.xxx|xxx.xxx.xxx.xxx>
[ERROR] Unable to detect kernel release on node XXX.

when running the script mentioned here https://longhorn.io/docs/1.4.1/deploy/install/#installation-requirements I have followed all the instructions listed before that.

big-judge-33880

04/20/2023, 8:48 AM

is there a decent way of mapping say

sdt

on a node to its associated longhorn volume?

big-judge-33880

04/20/2023, 10:32 AM

I’m struggling a bit with this getting spammed every 30sec in my kernel log:

Copy code

[Thu Apr 20 10:32:04 2023] blk_update_request: critical medium error, dev sdt, sector 14776192 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[Thu Apr 20 10:32:04 2023] sd 16:0:0:1: [sdt] tag#104 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Thu Apr 20 10:32:04 2023] sd 16:0:0:1: [sdt] tag#104 Sense Key : Medium Error [current]
[Thu Apr 20 10:32:04 2023] sd 16:0:0:1: [sdt] tag#104 Add. Sense: Unrecovered read error
[Thu Apr 20 10:32:04 2023] sd 16:0:0:1: [sdt] tag#104 CDB: Read(10) 28 00 00 e1 77 80 00 00 08 00
[Thu Apr 20 10:32:04 2023] blk_update_request: critical medium error, dev sdt, sector 14776192 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Thu Apr 20 10:32:04 2023] Buffer I/O error on dev sdt1, logical block 1846768, async page read
[Thu Apr 20 10:32:04 2023] Buffer I/O error on dev sdt2, logical block 0, async page read
[Thu Apr 20 10:32:04 2023] Buffer I/O error on dev sdt2, logical block 0, async page read
[Thu Apr 20 10:32:04 2023] Buffer I/O error on dev sdt2, logical block 0, async page read
[Thu Apr 20 10:32:04 2023] Buffer I/O error on dev sdt2, logical block 0, async page read
[Thu Apr 20 10:32:04 2023] Buffer I/O error on dev sdt2, logical block 0, async page read
[Thu Apr 20 10:32:34 2023] scsi_io_completion_action: 26 callbacks suppressed

hundreds-airport-66196

04/20/2023, 4:19 PM

Hi Rancher Team, For "Pod Deletion Policy When Node is Down", if I set this to "delete-statefulset-pod", it states here "Longhorn will force delete StatefulSet terminating pods on nodes that are down to release Longhorn volumes so that Kubernetes can spin up replacement pods." If the statefulset and volume are in the same node, will the new replacement pod connect to a replica volume? Assuming that the statefulset has replicas

late-needle-80860

04/23/2023, 6:19 PM

Have anyone seen this > https://github.com/longhorn/longhorn/discussions/5593?

👀 1

bright-wall-24452

04/24/2023, 7:51 AM

Hi Rancher Team,

bright-wall-24452

04/24/2023, 7:56 AM

Hi Rancher Team I deploy harvester with six nodes. My host network speed is faster than VM. I am trying to modify engine pod resource to 20% percent,but it's useless.I want to know How to improve VMs performance.

bitter-tailor-6977

04/24/2023, 8:03 PM

Hi All, i have installed longhorn in k3s node and uninstalled and tried to reinstall again in same node… i can see below error.. i have properly deleted / uninstalled as well…still issue persists logs from longhorn-manager-988w4 pod

Copy code

required setting default-share-manager-image shouldn't be empty" node=prodquartic
E0424 19:59:02.718884       1 volume_controller.go:226] failed to sync longhorn-system/pvc-52f24223-1029-4812-b10a-0c388a453c69: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> "ei-fc06c6fb" not found
E0424 19:59:02.718909       1 volume_controller.go:226] failed to sync longhorn-system/pvc-0d44dadf-a3df-4842-949c-84a3f688645d: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> "ei-fc06c6fb" not found
time="2023-04-24T19:59:02Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-52f24223-1029-4812-b10a-0c388a453c69 out of the queue" controller=longhorn-volume error="failed to sync longhorn-system/pvc-52f24223-1029-4812-b10a-0c388a453c69: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> \"ei-fc06c6fb\" not found" node=prodquartic
E0424 19:59:02.718887       1 volume_controller.go:226] failed to sync longhorn-system/pvc-0276039c-c505-411c-842e-941ad5213db1: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> "ei-fc06c6fb" not found
time="2023-04-24T19:59:02Z" level=warning msg="Dropping Longhorn volume longhorn-system/pvc-0d44dadf-a3df-4842-949c-84a3f688645d out of the queue" controller=longhorn-volume error="failed to sync longhorn-system/pvc-0d44dadf-a3df-4842-949c-84a3f688645d: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> \"ei-fc06c6fb\" not found" node=prodquartic
E0424 19:59:02.720284       1 volume_controller.go:226] failed to sync longhorn-system/pvc-a405ca9d-4778-4b49-a1b1-75ccaa00fd33: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> "ei-fc06c6fb" not found
E0424 19:59:02.720296       1 volume_controller.go:226] failed to sync longhorn-system/pvc-de46d2c9-2529-428d-bb33-4ecbd3172589: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> "ei-fc06c6fb" not found

E0424 19:59:02.772007       1 volume_controller.go:226] failed to sync longhorn-system/pvc-b6610285-2d44-4203-87d8-3fe524bf8047: unable to get engine image longhornio/longhorn-engine:v1.4.0: <http://engineimage.longhorn.io|engineimage.longhorn.io> "ei-fc06c6fb" not found
time="2023-04-24T19:59:02Z" level=fatal msg="Error starting manager: admission webhook \"<http://validator.longhorn.io|validator.longhorn.io>\" denied the request: failed to set the setting taint-toleration with invalid value : cannot modify toleration setting before all volumes are detached"

bright-wall-24452

04/25/2023, 5:36 AM

Hi Rancher Team！ When I attach a volume to node, the volume status is attaching. Could someone tell me how to check it?

quick-lawyer-7096

04/25/2023, 7:18 AM

Hi - This page https://longhorn.io/docs/1.4.1/monitoring/alert-rules-example/ provides a sample alert LonghornVolumeActualSpaceUsedWarning that reports when volumes are using more than 90% of there configured size - however this rule does not make sense to me as my volumes regularly use more than the configured space due to the snapshots. - this is well documented here https://longhorn.io/docs/1.4.1/volumes-and-nodes/volume-size/#volume-actual-size - have I missed something or is that sample alert wrong and miss-leading?

refined-analyst-8898

04/25/2023, 12:00 PM

I'm excited to discover Longhorn. It seems like the answer to complexity I've encountered with solving cluster storage. Looking forward to storage tiering improvements in Harvester, and have tiering working currently with guest clusters. Looking forward, what are the limitations of Longhorn for scale? Are there well known thresholds or boundaries I should be aware of so I can prepare to adapt topology or links?

stale-egg-64993

04/27/2023, 6:44 PM

Quick (hopefully) question, I'm looking to deploy longhorn on 2 specific nodes on a 10 worker-node cluster. I came across https://longhorn.io/kb/tip-only-use-storage-on-a-set-of-nodes/#create-a-storageclass-that-select-a-specific-set-of-nodes But one thing seemed a bit confusing/concerning:

Result: workloads that use Longhorn volumes can run on any nodes. Longhorn only schedules replicas of
my-longhorn-volv-pvc
on the node
node-1
,
node-2
, and
node-3

would that mean that potentially (in my case), workloads would only have a 20% chance of getting scheduled on a node for which longhorn is running (and by extension, where the PVs are)? 1. am I reading this right? 2. does this kb still apply given links in it go to v1.2.2 docs?

refined-analyst-8898

04/28/2023, 12:51 AM

@refined-analyst-8898 has left the channel

👋 1

fancy-car-52195

04/28/2023, 11:53 AM

Hello there, I'm getting a lot of these errors preventing me to retain volume backups... is there something that I could do ?

sparse-fireman-14239

05/01/2023, 11:49 AM

Is there a Longhorn resource I can query to see if Longhorn is up and running and at least somewhat sane? I'd prefer not to query Prometheus. I "need" this because I am deploying using Ansible (don't ask why, it is what it is 😄 )

fancy-car-52195

05/01/2023, 6:01 PM

I'm looking for a freelance longhorn specialist to setup some configurations on my cluster, is there anyone here?

late-needle-80860

05/09/2023, 2:51 PM

What would be the best way of verifying that the service behind the validator.longhorn.io webhook is up and healthy? I need to do this throughout the process of following the best practice of removing a Kubernetes worker node that’s also a longhorn node. And this is further needed because I’m removing nodes in the process of performing a rolling cluster upgrade … so one node goes ( drained and so on ) … and then the service behind the validator.longhorn.io webhook might not be fully ready on a new node. Yes I’m also filtering out the needed longhorn workloads from the drain …. Thank you 🤜🏽 🤛🏽 🙏

late-needle-80860

05/09/2023, 7:16 PM

I’m trying to identify what’s using the old engine-image on a cluster. Running Longhorn v1.4.1. The ref. count is two/2. I’ve controlled: • no volumes are using the old engine image • no volumes are upgradable to the v1.4.1 engine image • there’s no glaring log entries in syslogs on the nodes or in the longhorn pods What’s the way to identify the entities using the old engine image of v1.4.0?

ancient-pizza-13099

05/11/2023, 8:36 AM

is longhorn doing raid0 between nodes or is the maxium volume claim < biggest drive? Posted in #harvester

alert-policeman-61846

05/11/2023, 10:40 AM

Hello friends. Im looking for limits (max) on number of pv per host. My context is small cluster (3 workers) with huge capacity (bare metal servers). I want to use it for high density small apps hosting (hundreds of pods and longhorn pvc). Is there a limit i can set / raise on longhorn CSI driver ? Any OS/System config to help raising the limit (i understand each volume consumption will result in a mounted iscsi drive, guess there must be some limitation ?) Thks for any insight on that point