note it show s its attached to tow instance managers i ve tr rancher-users #harvester

Title

# harvester

quaint-alarm-7893

02/10/2023, 6:19 AM

note, it show's its attached to tow instance managers. i've tried rebooting the nodes, and deleting the instance manager pods, no luck...

great-bear-19718

02/13/2023, 6:00 AM

does this pod still exist? virt-launcher-vmname-mpbn2?

quaint-alarm-7893

02/13/2023, 11:26 PM

@great-bear-19718 no. it's gone.

just posted this issue in longhorn too: https://github.com/longhorn/longhorn/issues/5388

any help would be appreciated.

great-bear-19718

02/13/2023, 11:40 PM

random question but have you rebooted this node?

kubectl get node NODENAME -o yaml

quaint-alarm-7893

02/13/2023, 11:40 PM

yes

i rebooted both nodes the instance is on, and the node holding the replica

great-bear-19718

02/13/2023, 11:41 PM

👍

yeah i just saw in the issue

are you please also able to generate a harvester support bundle?

or provide me one if you have already generated it

quaint-alarm-7893

02/13/2023, 11:51 PM

sure, one sec.

supportbundle_9042f568-6514-4a9f-a6c3-96a342641671_2023-02-13T23-50-55Z.zip

great-bear-19718

02/14/2023, 12:26 AM

the pvc is gone

oh wait its in

vdi

namespace

quaint-alarm-7893

02/14/2023, 12:27 AM

correct

i was gunna say. lol

great-bear-19718

02/14/2023, 12:27 AM

are you able to go to harvester settings and there is a setting for support-bundle-namespaces

or additional namespaces..

add

vdi

to it and regenerate the support bundle?

Copy code

Kubernetes Status:
    Last Pod Ref At:  2023-02-12T04:21:48Z
    Namespace:        vdi
    Pv Name:          pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a
    Pv Status:        Bound
    Pvc Name:         ivanquezada-rootdisk-ossjw
    Workloads Status:
      Pod Name:          virt-launcher-ivanquezada-k8phc
      Pod Status:        Pending
      Workload Name:     ivanquezada
      Workload Type:     VirtualMachineInstance

its waiting for the pod to be launched it seems

quaint-alarm-7893

02/14/2023, 12:28 AM

i agree. but it's not launching, and if i launch it, it just hangs.

great-bear-19718

02/14/2023, 12:28 AM

another support bundle with the additional namespace would be useful

quaint-alarm-7893

02/14/2023, 12:28 AM

regenerating now

great-bear-19718

02/14/2023, 12:29 AM

thanks

also seems like you only have 1 replica left

for this volume compared to the default 3

quaint-alarm-7893

02/14/2023, 12:29 AM

yeah, i tried that too to see if it would dismount

didnt work.

supportbundle_9042f568-6514-4a9f-a6c3-96a342641671_2023-02-14T00-28-51Z.zip

great-bear-19718

02/14/2023, 1:04 AM

i can see the error.. but no two snapshot classes with same name, i have asked the longhorn team as well

quaint-alarm-7893

02/14/2023, 1:05 AM

okay. anything i can do on my side?

great-bear-19718

02/14/2023, 1:05 AM

not yet.. i will let you know

quaint-alarm-7893

02/14/2023, 1:05 AM

great. thanks!

great-bear-19718

02/14/2023, 1:13 AM

Copy code

(⎈|default:longhorn-system)➜  harvester-758d56647d-wlgj2 k get engine | grep pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a
pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a-e-4596c93c   error     harvester-05   instance-manager-e-c2f010c0                                       4d7h
pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a-e-6591f94e   error     harvester-03   instance-manager-e-50d28d2d                                       5d

i see the two engine instances

i am just checking if one can be deleted safely

quaint-alarm-7893

02/14/2023, 6:23 PM

@great-bear-19718 did you happen to find out anything?

great-bear-19718

02/14/2023, 9:59 PM

yeh you should be able to delete one of the engine objects it seems

kubectl delete engine pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a-e-6591f94e -n longhorn-system

quaint-alarm-7893

02/14/2023, 11:59 PM

so that released it. now it's showing as faulted. but there are no replicas to salvage it to.

great-bear-19718

02/15/2023, 12:00 AM

yeah all the replicas were unfortunately unhealthy

quaint-alarm-7893

02/15/2023, 12:00 AM

so is there a way to get it to mount?

great-bear-19718

02/15/2023, 12:00 AM

what does

kubectl get replicas -n longhorn-system

show?

quaint-alarm-7893

02/15/2023, 12:01 AM

nothing for that pvc

it does still show up a a pvc, and a pv. so is there any way to get it mounted into a vm? or not really?

great-bear-19718

02/15/2023, 12:03 AM

Copy code

kubectl get replica -n longhorn-system | grep pvc-13787ebc-7d4e-41ad-9206-de9a6adb938

i think this was the last replica for the pv in question

quaint-alarm-7893

02/15/2023, 12:03 AM

nothing

great-bear-19718

02/15/2023, 12:06 AM

may be your best option https://longhorn.io/docs/archives/1.1.1/advanced-resources/data-recovery/export-from-replica/

if the replica crd is gone

quaint-alarm-7893

02/15/2023, 12:10 AM

okay, i'll take a stab at it a little later. thanks!

quick question. do you think if i didnt delete some of the replicas, and kept them, but just deleted the engine, i would have a replica to salvage still?

thank you for your help with this by the way. i appreciate it

3 Views