This message was deleted.

Hi @quaint-alarm-7893, could you provide the support bundle? We can check for it. BTW, are the problematic disks extra?

@salmon-city-57654 thanks! attached is the bundle. and yes, these are all extra disks.

Thanks @quaint-alarm-7893, I will take a look for this issue.

@salmon-city-57654 i ran:

k delete po harvester-node-disk-manager-vv7qw -n harvester-system

the pod removed and a new one spun up. i still cant add any disks back. using the edit storage / add disk option, and the no disks show up in longhorn.

Hi @quaint-alarm-7893, could you generate a SB again? I would like to check the failure after the node-disk-manager respawned

@salmon-city-57654 here you go.

Hi @quaint-alarm-7893, sorry for the late update here. I checked your first SB, and found the issue [ENHANCEMENT] Should respawn the udev monitor if any errors Then, but the the

StorageReserved

on your environment is

0

. This value should be updated by the longhorn-manager. I will check on the code level to find the root cause. So, is it still in the same situation in your environment?

yeah, same issue. basically i have a bunch of disks i cant re-configure and start using again.

Hi @quaint-alarm-7893, I checked your SB. I thought that because your

instance-manager-r

on the node2 is missing. Did you do any operations on node2? So you can not schedule any replicas to node2.

@salmon-city-57654 nothing i recall. but it looks like it's disabled in l.h. no disks show up.

Hi @quaint-alarm-7893, did you remember any operations on the node2? seems all disk statuses on the harvester-node2 are removed

spec:
    allowScheduling: true
    disks: {} <-- should not empty
    engineManagerCPURequest: 0
    evictionRequested: false
    name: harvester-02
    replicaManagerCPURequest: 0
    tags: []

I still saw some blockdevices CR on the harvester-node-2. Could you try to delete the node-disk-manager on the harvester-node-2?

i've deleted the pod, where can i check the disk statuses again?

check the

<http://nodes.longhorn.io|nodes.longhorn.io>

@salmon-city-57654 still shows the same, empty. i think when i was initially trying to get it to work, i deleted the disks in lh. maybe that's why? any way i can bring them back?

Can you try to check with the following command?

# kubectl get replicas |grep harvester-node-2

there are no replicas. can you tell me the command to remove blockdevice?

# kubectl get blockdevices -A |grep harvester-node-2

you should get all blockdevices on the node-2

okay, so i got the devices available to "add" now in the storage tab in harvester. but quick question. looks like when i cleared out everything, i also cleared the "default-disk" storage too. is that going to be an issue? or is it fine? i guess i just want to make sure if it's an issue or not, because at this point, i can just nuke the node and make a new one if that's a better solution.

default disk is not involved in blockdevices CR. (If you means the longhorn default disk after installation)

just /dev/sda - /dev/sdq

Hi @quaint-alarm-7893, I cannot find the related default disk on your node 2. Do you remember what operations after the previous time you saw the replica on the node2? Or you can move the VM away from node2 then kick out -> rejoin the node2? That will be more clean. Or you can skip the default disk. We still have other disks for the replicas

i deleted all disks in both longhorn and harvester when i was having the issues. i likely deleted the default disk too. is it needed to operate properly? or is it not really nessessary?

If you have other disks, the default disk can be ignored