This message was deleted.

Hi! What is the output of the the following command? Do you see anything strange?

kubectl -n longhorn-system get lhr

Yes i see al lot of pvc with stopped state

Those are the replicas. Can you show me the replica associated with the problematic node? Like this: kubectl -n longhorn-system get lhr ID -o yaml

apiVersion: longhorn.io/v1beta2 kind: Replica metadata: creationTimestamp: "2024-03-12T102855Z" generation: 1 labels: longhorn.io/backing-image: "" longhorndiskuuid: "" longhornnode: "" name: pvc-fe8c0606-efbb-4a32-9aa8-67b7a797b4bc-r-f8b5dea8 namespace: longhorn-system ownerReferences: - apiVersion: longhorn.io/v1beta2 kind: Volume name: pvc-fe8c0606-efbb-4a32-9aa8-67b7a797b4bc uid: 11c6a626-ad81-40b0-b4bd-85a6d1ce72a3 resourceVersion: "485485953" uid: 002761f5-d850-46ab-8817-9c823b0abfa0 spec: active: true backingImage: "" baseImage: "" dataDirectoryName: "" dataPath: "" desireState: stopped diskID: "" diskPath: "" engineImage: longhornio/longhorn-engine:v1.4.0 engineName: pvc-fe8c0606-efbb-4a32-9aa8-67b7a797b4bc-e-56db2b76 failedAt: "" hardNodeAffinity: "" healthyAt: "" logRequested: false nodeID: "" rebuildRetryCount: 5 revisionCounterDisabled: false salvageRequested: false unmapMarkDiskChainRemovedEnabled: false volumeName: pvc-fe8c0606-efbb-4a32-9aa8-67b7a797b4bc volumeSize: "10737418240" status: conditions: - lastProbeTime: "" lastTransitionTime: "2024-03-12T102855Z" message: "" reason: "" status: "True" type: InstanceCreation currentImage: "" currentState: stopped evictionRequested: false instanceManagerName: "" ip: "" logFetched: false ownerID: rr8-aw07 port: 0 salvageExecuted: false started: false storageIP: ""

Hm. And your PV has other healthy replicas? Or stucked in rebuilding phase?

yes

If it has other replicas and you have a backup My opinion is that you can delete the replica above. For example with kubectl delete lhr command

thanks, great help

wow 🙂

yes i know

maybe you can do something like this:

kubectl  -n longhorn-system  get lhr -l longhornvolume=pvc-d97403bd-0d61-40a4-8a97-b2e9db8439b2
kubectl  -n longhorn-system  delete lhr -l longhornvolume=pvc-d97403bd-0d61-40a4-8a97-b2e9db8439b2

i see these

what was your command ?

im going with this now -> kubectl -n longhorn-system get replica --selector longhornnode= | grep stopped | awk '{print $1}' | kubectl -n longhorn-system delete replica

try like this :

for i in $(kubectl  -n longhorn-system  get lhr | grep pvc-f19aed02-12b9-451f-8111-f54b5bc23019 | awk '{print $1}' ); do kubectl -n longhorn-system get lhr $i;  done

for i in $(kubectl -n longhorn-system get lhr --selector=longhornnode= | grep stopped | awk '{print $1}' ); do kubectl -n longhorn-system delete lhr $i; done

Deleting longhorn-manager is a safe operation. It should not cause anything interesting to happen on the data plane.

Check i figured that out, thanks for the confirmation

Do you have a support bundle from when you had thousands of replicas you are willing to share @polite-alarm-96476? The Longhorn team would be interested in trying to discover what particular race condition led to the creation of all of those unnecessary replicas.

Hi Eric, im afraid i have not, but the next time ( i hope not) i have some funky stuff i will create one.