hello everyone i have a vm in harvesteR that got messed up w rancher-users #longhorn-storage

Title

# longhorn-storage

quaint-alarm-7893

02/10/2023, 6:19 AM

hello everyone, i have a vm (in harvesteR) that got messed up when one of my servers had an issue, and now the VM is off, but the PVC is still attached. how can i detach it so i can bring the vm back online?

Copy code

State: Attached
Health:
No node redundancy: all the healthy replicas are running on the same node Degraded
Ready for workload:Not Ready
Conditions:
restore
scheduled
Frontend:Block Device
Attached Node & Endpoint:
Size:
100 Gi
Actual Size:Unknown
Data Locality:disabled
Access Mode:ReadWriteMany
Backing Image:vdi-image-hzwdh
Backing Image Size:50 Gi
Engine Image:longhornio/longhorn-engine:v1.3.2
Created:a month ago
Encrypted:False
Node Tags:
Disk Tags:
Last Backup:
Last Backup At:
Replicas Auto Balance:ignored
Instance Manager:
instance-manager-e-c2f010c0
instance-manager-e-50d28d2d
Last time used by Pod:10 hours ago
Namespace:vdi
PVC Name:vmname-rootdisk-ossjw
PV Name:pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a
PV Status:Bound
Revision Counter Disabled:False
Last Pod Name:virt-launcher-vmname-mpbn2
Last Workload Name:vmname
Last Workload Type:VirtualMachineInstance

note, it show's its attached to tow instance managers. i've tried rebooting the nodes, and deleting the instance manager pods, no luck...

narrow-egg-98197

02/11/2023, 3:00 AM

Do you setup etcd in multiple nodes? If yes, I think this case might same as this Issue#1755 .

quaint-alarm-7893

02/11/2023, 6:39 PM

@narrow-egg-98197 yes. it's harvester cluster with 6 nodes. one of the nodes is etcd member, one is not. can i just delete the pvc without losing the data? i guess that's my worry is if i delete the pvc, the data would be lost. sounds like that's the work around though.

@narrow-egg-98197 i've looked at that, i cant find anything mounting that pvc. i cant figure out how to "detach it" it's not an option available in the ui. i've shut off one node that was running the instance-manager, and i killed the pod on the other server, and it still wont release.

narrow-egg-98197

02/14/2023, 4:18 PM

can i just delete the pvc without losing the data? i guess that's my worry is if i delete the pvc, the data would be lost. sounds like that's the work around though.

Is it workable to create backup for this volume then redeploy related workloads, e.g., Longhorn volume, PVC, PV and Pods?

quaint-alarm-7893

02/14/2023, 4:20 PM

with a little guidance, yes. not sure how to back up the pvc, and i just use dd and give it the path of the pvc, and the path of a new pvc that's blank? just like if i were cloning a disk or partition.

also, a harvester team member was also helping w/ this. he said this yesterday, do you maybe know the answer? he hasnt gotten back to me yet. rancher employee [6:13 PM]

Copy code

(⎈|default:longhorn-system)➜  harvester-758d56647d-wlgj2 k get engine | grep pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a
pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a-e-4596c93c   error     harvester-05   instance-manager-e-c2f010c0                                       4d7h
pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a-e-6591f94e   error     harvester-03   instance-manager-e-50d28d2d                                       5d

i see the two engine instances gauravm [6:13 PM] i am just checking if one can be deleted safely

@narrow-egg-98197 ^

24 Views