https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
m
To note, the VM does get stuck in starting but I did the workaround in the upgrade notes and it didn't work. It is still missing a node in Longhorn for the volume, which is attempting to attach to a node that doesn't have a replica.
b
Which work around did you try, clearing currentMigrationNodeID ?
m
yeah
b
did it get repopulated after you cleared it?
m
unfortunately, no
the volume is flopping between attaching/unattaching still
even cleared out the instance-manager on the node that doesn't have a replica
b
Does the volume have multiple attachmentTickets associated with it?
trying to find the github issue with details for that
m
where can I see the attachmentTickets for that volume?
b
kubectl -n longhorn-system get <http://VolumeAttachment.longhorn.io|VolumeAttachment.longhorn.io> pvc-ID -o yaml
m
I only see one
b
if you do 
kubectl -n longhorn-system get <http://Volume.longhorn.io|Volume.longhorn.io> pvc-ID -o yaml
, is 
nodeID
set to the node not in the replica list?
m
yes
b
could you try shutting down the VM, clearing that nodeID (or maybe clearing the nodeID and then shutting down VM), and seeing if it comes back to life?
m
the VM is shut down because it wasn't booting, I did it before when it was shut down but I'll do it again
oh wait, migrate status, yes. so what's the command to clear nodeid?
b
hmm I am trying to remember now which property we edited when we had this is issue, I think it may have been 
PendingNodeID
rather than 
nodeId
. Either way you can do a 
kubectl -n longhorn-system edit <http://Volume.longhorn.io|Volume.longhorn.io> pvc-ID
to set it to an empty string
wait was the volume attachment ticket you saw related to longhorn UI?
m
I was just looking in longhorn UI at what was going on with the volume, more than anything.
b
Sorry I more meant when you ran 
kubectl -n longhorn-system get <http://VolumeAttachment.longhorn.io|VolumeAttachment.longhorn.io> pvc-ID -o yaml
and saw 1 ticket, was it called csi- something or was it called something with longhorn UI in the name
m
Copy code
spec:
  attachmentTickets:
    longhorn-ui:
      generation: 0
      id: longhorn-ui
      nodeID: frac-harvint-03
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: longhorn-api
  volume: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
b
Ah good, if you delete that entire longhorn-ui block using kubectl edit, it should work
m
k
b
m
just to be sure - the only thing under spec should be the volume: pvc-id ?
b
leave attachmentTickets:
Copy code
spec:
  attachmentTickets: {}
  volume: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
m
ok
so, now should I try starting the vm?
b
yeah go for it
m
still having an issue -
Copy code
AttachVolume.Attach failed for volume "pvc-5ebec812-07b4-46af-ad8c-c43162b02264" : rpc error: code = DeadlineExceeded desc = volume pvc-5ebec812-07b4-46af-ad8c-c43162b02264 failed to attach to node frac-harvint-03 with attachmentID csi-8068e3962fe65255e3d22679f7f7ad69048d0d625f22109bdcdc081393da5a02
b
and if you turn it off again, clear the attachment tickets, does the Volume have 
PendingNodeID
set
from 
kubectl -n longhorn-system edit <http://Volume.longhorn.io|Volume.longhorn.io> pvc-ID
m
I just powered it off, didn't run any other commands and this is the yaml
Copy code
apiVersion: <http://longhorn.io/v1beta2|longhorn.io/v1beta2>
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-08-25T17:13:53Z"
  finalizers:
  - <http://longhorn.io|longhorn.io>
  generation: 37
  labels:
    longhornvolume: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
  name: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: <http://longhorn.io/v1beta2|longhorn.io/v1beta2>
    kind: Volume
    name: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
    uid: dcad0313-753e-49b2-b3fd-40913035d7c7
  resourceVersion: "159714947"
  uid: 1957a878-19ea-4977-af25-1aafea02d800
spec:
  attachmentTickets: {}
  volume: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
status:
  attachmentTicketStatuses: {}
b
please can you also show 
kubectl -n longhorn-system get <http://Volume.longhorn.io|Volume.longhorn.io> pvc-5ebec812-07b4-46af-ad8c-c43162b02264 -o yaml
m
Copy code
apiVersion: <http://longhorn.io/v1beta2|longhorn.io/v1beta2>
kind: Volume
metadata:
  creationTimestamp: "2025-08-25T17:13:53Z"
  finalizers:
  - <http://longhorn.io|longhorn.io>
  generation: 34041
  labels:
    backup-target: default
    longhornvolume: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
    <http://recurring-job-group.longhorn.io/default|recurring-job-group.longhorn.io/default>: enabled
    <http://setting.longhorn.io/remove-snapshots-during-filesystem-trim|setting.longhorn.io/remove-snapshots-during-filesystem-trim>: ignored
    <http://setting.longhorn.io/replica-auto-balance|setting.longhorn.io/replica-auto-balance>: ignored
    <http://setting.longhorn.io/snapshot-data-integrity|setting.longhorn.io/snapshot-data-integrity>: ignored
  name: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
  namespace: longhorn-system
  resourceVersion: "159715018"
  uid: dcad0313-753e-49b2-b3fd-40913035d7c7
spec:
  Standby: false
  accessMode: rwx
  backingImage: ""
  backupCompressionMethod: lz4
  backupTargetName: default
  dataEngine: v1
  dataLocality: disabled
  dataSource: <vol://pvc-0abc17c0-3bc5-42e1-8528-4b95c7d67c74>
  disableFrontend: false
  diskSelector: []
  encrypted: false
  freezeFilesystemForSnapshot: ignored
  fromBackup: ""
  frontend: blockdev
  image: longhornio/longhorn-engine:v1.9.2
  lastAttachedBy: ""
  migratable: true
  migrationNodeID: ""
  nodeID: ""
  nodeSelector: []
  numberOfReplicas: 3
  offlineRebuilding: ignored
  replicaAutoBalance: ignored
  replicaDiskSoftAntiAffinity: ignored
  replicaSoftAntiAffinity: ignored
  replicaZoneSoftAntiAffinity: ignored
  restoreVolumeRecurringJob: ignored
  revisionCounterDisabled: true
  size: "128849018880"
  snapshotDataIntegrity: ignored
  snapshotMaxCount: 250
  snapshotMaxSize: "0"
  staleReplicaTimeout: 30
  unmapMarkSnapChainRemoved: ignored
status:
  actualSize: 35117473792
  cloneStatus:
    attemptCount: 1
    nextAllowedAttemptAt: "2025-08-25T17:14:23Z"
    snapshot: deb4f8dc-9856-582f-aca9-3d53bebd1dc7
    sourceVolume: pvc-0abc17c0-3bc5-42e1-8528-4b95c7d67c74
    state: completed
  conditions:
  - lastProbeTime: ""
    lastTransitionTime: "2025-08-25T17:13:53Z"
    message: ""
    reason: ""
    status: "False"
    type: WaitForBackingImage
  - lastProbeTime: ""
    lastTransitionTime: "2025-08-25T17:13:53Z"
    message: ""
    reason: ""
    status: "False"
    type: TooManySnapshots
  - lastProbeTime: ""
    lastTransitionTime: "2025-08-25T17:13:53Z"
    message: Reset schedulable due to allow volume creation with degraded availability
    reason: ""
    status: "True"
    type: Scheduled
  - lastProbeTime: ""
    lastTransitionTime: "2025-08-25T17:13:53Z"
    message: ""
    reason: ""
    status: "False"
    type: Restore
  currentImage: longhornio/longhorn-engine:v1.9.2
  currentMigrationNodeID: ""
  currentNodeID: ""
  expansionRequired: false
  frontendDisabled: false
  isStandby: false
  kubernetesStatus:
    lastPVCRefAt: ""
    lastPodRefAt: "2025-11-17T19:54:25Z"
    namespace: default
    pvName: pvc-5ebec812-07b4-46af-ad8c-c43162b02264
    pvStatus: Bound
    pvcName: frac-ubnt2204-02-rootdisk-fd74r
    workloadsStatus:
    - podName: virt-launcher-frac-ubnt2204-02-58lbk
      podStatus: Failed
      workloadName: frac-ubnt2204-02
      workloadType: VirtualMachineInstance
  lastBackup: backup-8e305a3be8924107
  lastBackupAt: "2025-09-16T18:02:36Z"
  lastDegradedAt: ""
  ownerID: frac-harvint-03
  remountRequestedAt: "2025-11-17T19:54:41Z"
  restoreInitiated: false
  restoreRequired: false
  robustness: unknown
  shareEndpoint: ""
  shareState: ""
the frac-harvint-03 is the one that is having an issue since it isn't listed in Longhorn as having a replica
b
hmm, running out of ideas. When the VM starts, is is trying to start on frac-harvint-03? Could you set a node selector for the VM to another node
in the console under 
Node Scheduling
for the VM
m
it is trying to start on 03, let me see if I can change that in scheduling
so, I forced it to start on my 05 node, and I see in longhorn it created a replica on the 03 node ... why the even ....
but, it started, so I guess I cannot complain
🎉 1
b
must be some CRD or something floating around that really likes node 03
m
and now the replica on 03 just disappeared ... and then just came back up. so it is stuck in degraded
why is it so stuck up on putting a replica on this one node
b
does it have the most disk space free? Are you still stuck mid harv upgrade?
m
it does, and I am stuck mid upgrade since it will not reboot a node if it has degraded volumes
b
can you get to the embedded longhorn UI in harv console?
m
I am, I am looking at it now.
volumes show green in harvester but there's degraded volumes in longhorn
b
in the nodes tab, can you select the problematic node, edit it, disable scheduling
that should prevent node volumes being created on it
m
ok, I disabled scheduling but ... that won't be the solution long term I take it, once the migration is completed?
b
no hopefully this just allows it to reboot and the upgrade to progress
and hopefully a reboot fixes whatever is wrong with it
m
well, the funny thing is, the 03 node was already upgraded
but now it looks like it is doing some rebuilds
I believe something is wrong with longhorn on that one node
b
seems like it, maybe another reboot will kick it into life afterwards but wouldn't advise that mid upgrade
hopefully this is enough to progress the upgrade
m
thanks for the help - I'll let these volumes replicate and rebuild, hopefully it will go smooth. I'll be back if I'm stuck on anything. Thanks Huw!
b
np. Hopefully all goes well, I am off for the evening now but will be back tomorrow
(before I forget - also want to remove the node selector from that one VM when everything is working because that will prevent live migrations in the future)
m
will do
Howdy @brainy-kilobyte-33711 - so, looks like my node still has not rebooted for the upgrade. Unscheduling that one host prevented some of my volumes from replicating, so honestly I set them all to 2 replica to prevent any degradation health issues. I couldn't run any kubectl commands on the node that needed to be rebooted, and it appeared the $KUBECONFIG was empty - so I copied the file from another node in the cluster and that started working but it appears the instance manager is still errored, which I cannot kill since it isn't active.
b
I think anything I suggest has a large chance to make things worse so I'd recommend creating a github issue and attaching the support bundle. Hopefully someone from SUSE can help out.
m
Alrighty, well thanks for your help! Just curious, what other options would there potentially be?
b
killing pods and manually restarting nodes until the issue is resolved but doing that mid upgrade (or any point really) has the potential to nuke the entire cluster. The support bundle with logs will help isolate where the issues are and allow a more surgical fix
maybe someone else reading this thread has experienced the issue and might have suggestions
m
That's where I've been - trying to kill pods that seem to be hanging stuff up. I'll go the Github route, though.
12 Views
Open in Slack
PreviousNext
Powered by