This is a paid analysis of the longhorn + k3s deployment project

the K3s ( metrics-server, coredns ) and another process TGT goes super highest CPU usage

Hey there! ICYMI - the Longhorn team release v1.4 just before the new year. If you want a quick run through of what some of the enhancements are - check out this blog https://www.suse.com/c/rancher_blog/longhorn-1-4-starting-a-new-year-with-a-new-release/

@witty-tiger-89786 -- are you able to provide a commercial support for K3s + longhorn deployment?

I seriously frustrated to use this longhorn and K3s ( as opposed to their promises of Low Resources and Stability ) they are completely opposite

On a customer cluster, the customer use Zabbix for OS monitoring and to keep an eye on disk usage/perf on /var/lib/longhorn. That works well, the issue is that they receive response time alarms on longhorn volumes, but seeing as Lognhorn is using the Linux SCSI subsystem and get dynamically assigned sdX devices, I struggle to find a good way to differentiate between what are OS and Longhorn disks. Anyone else struggle with this, and have found a solution?

Hi! Based on the Longhorn documentation (https://longhorn.io/docs/1.4.0/volumes-and-nodes/scheduling/#the-node-and-zone-selection-stage), I understand that it is possible for Longhorn to create replicas by creating replicas of a volume so that they are placed on separate sites. For example, if I'm using a three-sites design, with a setting of "defaultClassReplicaCount: 3", each site will have one replica. My question is, how can I tell Longhorn which node belongs to which site? Can the Kubernetes zone tag (https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) be suitable for this, or does it need to be used in a separate tag for Longhorn?

Hi all. I've got a weird issue trying to install Longhorn on an RKE2 cluster via the Helm chart. My

longhorn-manager-*

pods aren't coming up and instead going in a

CrashLoopBackOff

status. Running

kubectl logs pod/longhorn-manager-xxxxx -n longhorn-system

gives me:

Defaulted container "longhorn-manager" out of: longhorn-manager, wait-longhorn-admission-webhook (init)
W0119 13:13:36.447109       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2023-01-19T13:13:36Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n: exit status 2"
I0119 13:13:36.496292       1 leaderelection.go:248] attempting to acquire leader lease longhorn-system/longhorn-manager-upgrade-lock...
I0119 13:13:36.510495       1 leaderelection.go:258] successfully acquired lease longhorn-system/longhorn-manager-upgrade-lock
time="2023-01-19T13:13:36Z" level=info msg="Start upgrading"
time="2023-01-19T13:13:36Z" level=info msg="setting default-engine-image not found"
time="2023-01-19T13:13:46Z" level=error msg="Upgrade failed: upgrade API version failed: cannot create CRDAPIVersionSetting: Internal error occurred: failed calling webhook \"<http://validator.longhorn.io|validator.longhorn.io>\": failed to call webhook: Post \"<https://longhorn-admission-webhook.longhorn-system.svc:9443/v1/webhook/validaton?timeout=10s>\": context deadline exceeded"
time="2023-01-19T13:13:46Z" level=info msg="Upgrade leader lost: <http://c3.hq.iwink.nl|c3.hq.iwink.nl>"
time="2023-01-19T13:13:46Z" level=fatal msg="Error starting manager: upgrade API version failed: cannot create CRDAPIVersionSetting: Internal error occurred: failed calling webhook \"<http://validator.longhorn.io|validator.longhorn.io>\": failed to call webhook: Post \"<https://longhorn-admission-webhook.longhorn-system.svc:9443/v1/webhook/validaton?timeout=10s>\": context deadline exceeded"

csi-resizer

in continuous crashloopback. any pointer. attaching logs for reference.

Hi. I recently installed longhorn 1.34 on my k3s cluster. I have 3 master nodes, one of them is tainted with NoSchedule (node.kubernetes.io/unschedulable:NoSchedule) I still see this node when I go to longhorn ui -> Nodes. It appears as not scedhulable but it makes me unable to create backups to NFS and all sort of weird issues. The only way to get rid of that node is to remove the CR that represents it directly, but it will come back at some point (and since its not schedulable, ill have instance managers that are unable to deploy on it). Any suggestions why it happens? I also have "*Disable Scheduling On Cordoned Node*" checked.

API server is overwhelmed and causing intermittent timeout due to rancher. Any pointer how to solve this? K8s: EKS 1.21+ Rancher: 2.7

i had a longhorn volume go into a degraded state last night - second time in the last few days. it recovered quickly, but i'm curious if there is something i should dig into further doesn't look like i have any hardware issues on the host, when digging into logs, i found that a csi-provisioner pod on the same host crashed

Has anyone seen the following error when spinning up Longhorn? I'm starting it from Rancher (itself spun up from latest) via the Helm chart. I'll put a few extra details in thread, as I've fiddled with it a bit.

spec: failed to generate spec: path "/var/lib/longhorn/" is mounted on "/" but it is not a shared mount

@orange-airplane-98016 has left the channel

Looking for information on how to implement an Opensearch/Elasticsearch hot/warm/cold infrastructure using Longhorn storage in a Rancher RKE environment. I'm not sure if this is solved by Longhorn or if it's solved in the k8s configuration. If anyone can point me in the right direction, it would be much appreciated.

Hello friends. I recently added longhorn 1.4 on a multi-az and multi-region k3s cluster. Works like a charm. I tested the "best-effort" data locality. Im wondering if best effort will take care of regions (from the doc, seems node colocation and az are taken care off, not sure about regions https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion). https://longhorn.io/docs/1.4.0/high-availability/data-locality/ My ideal setup would place active replica on same node as pod, and replicas in other nodes in same az. To avoid inter-region replication traffic. Is this possible ?

Any word on when Longhorn 1.4.x will be available through Rancher Manager?

On a small cluster with 3 worker nodes, and each worker node has a Longhorn replica, when I use the system-upgrade controller to patch Ubuntu and reboot, sometimes it's too fast and one or several volumes are still degraded when it goes ahead and schedules the next worker node, leaning me with one intact replica and two failed. Obviously not ideal 🙂 So, would a good solution be, to use a "prepare init container" in the system-upgrade plan, and have it check and wait for all volumes to be OK before proceeding? Or is there a better way?

Just made a new cluster on rancher with RKE2 and I can’t see Longhorn in the apps section

Has it been removed?

How can I get the latest Longhorn 1.4.0 to appear under Charts in Rancher?

hello everyone, I couldn't enter the longhor slack, so found this channel here, Ill post my problem here then, maybe somehone has any clues I have a pvc inside longhorn, that is in a loop state attaching / detaching / remount and the pods needing this data cannot be created. Is there any way to troubleshoot this or stop the loo proccess so I can for instance attach it without the pod and create a new replica to delete one of them?

image.png

image.png

image.png

it keeps alternating on those 2 states, cannot do nothing on the interface

I’ve been debating dedicating an LVM2 group to containerd and kubelet data … in this issue: https://github.com/k3s-io/k3s/issues/2068 … I’ve followed Mr. @*mdrakiburrahman* suggestions in that issue and also

symlinked

/var/lib/kubelet to the dedicated disk >> this to get all Pod related data onto that disk. Now I’m unfortunately seeing, for workloads using a PVC’s, the following err:

MountVolume.SetUp failed for volume "pvc-7dc11d73-3595-47a9-bb02-a95f23518ca5" : applyFSGroup failed for vol pvc-7dc11d73-3595-47a9-bb02-a95f23518ca5: lstat /k3s-worker-data/kubelet/pods/12bfddbb-a8fe-4edb-9620-6e96f40ce840/volumes/kubernetes.io~csi/pvc-7dc11d73-3595-47a9-bb02-a95f23518ca5/mount: no such file or directory

If I create the

mount

directory manually files are created in the dir … However, comparing the permissions on the dir with permissions on another workload running on a cluster where I’m NOT symlinking to dedicate a disk to container data, the permissions are different. On the non-working node it’s the

fsGroup

on the files in sub-dirs to

mount

. I’m in doubt in regards to how to set these permissions, if at all I need to. The Stateful workload I’m troubleshooting do set the

fsGroup

in the

SecurityContext

in it’s

Kind: StatefulSet

manifest. Any ideas? Suggestions are VERY WELCOME

How can I force delete a nodes.longhorn.io node object/node … even if I scale back the node into the K3s cluster ( v1.25.6+k3s1 ) - I still can’t remove the node. From neither the Longhorn UI or by using

kubectl delete …

directly. Any ideas and thank you

Is there a way to automatically remove claimRef from PersistentVolume after PVC has been deleted? I’m trying to create a pool of reusable cache volumes, but the problem is that claimRef prevents reuse of PVs after pod and its PVC have been deleted.

Hello to all. I wanted to get your opinion on Longhorn as a distributed filesystem for private cloud production environments. What experience do you have? Many thanks.