https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# longhorn-storage
a
This message was deleted.
p
Try the upgrade again?
n
Well the upgrade never failed in the first place
the cluster seems to think 1.7.1 is installed properly
p
Yeah but you can still upgrade an app in place
n
allright, will try
p
Just do reinstall/upgrade -> next -> next
It will be basically a redeploy
n
Im worried about hitting the same problem though
p
It cant be worse anyway no?
n
It worked, but it didnt seem to change anything (i.e. I still see longhorn components versioned at 1.6.2)
Im also hitting this now:
Copy code
MountVolume.MountDevice failed for volume "pvc-7382a499-3d90-477d-a095-5e476e95b0cd" : rpc error: code = Internal desc = mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-7382a499-3d90-477d-a095-5e476e95b0cd /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/7679414816d052a2e817f12e9023f7cdc0eb6094ff830b369ee09fb6f8a72513/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/7679414816d052a2e817f12e9023f7cdc0eb6094ff830b369ee09fb6f8a72513/globalmount: /dev/longhorn/pvc-7382a499-3d90-477d-a095-5e476e95b0cd already mounted or mount point busy. dmesg(1) may have more information after failed mount system call.
p
huh
n
Any idea @powerful-librarian-10572? I'm pretty stumped myself at this point
p
this might be dangerous, by you can try (emphasis on try, as it may explode everything, altough it didnt cause issue on my test cluster) to redeploy the longhorn-manager daemonset
n
Ultimately I think the issue is with lingering 1.6.2 components alongside 1.7.1 components, would you agree?
Perhaps 1.7.1 can't fully take over since 1.6.2 is still being used somewhere by the system
p
Probably, altough having images engine of 1.6.2 will not cause any issues
What components are stuck on 1.6.2 ?
n
rancher/mirrored-longhornio-longhorn-manager:v1.6.2 rancher/mirrored-longhornio-longhorn-manager:v1.6.2 (twice on the same node apparently?) rancher/mirrored-longhornio-longhorn-engine:v1.6.2 rancher/mirrored-longhornio-longhorn-instance-manager:v1.6.2
I imagine its the instance manager causing most problems
p
yes and the instance manager is a pod so no clue
n
wdym?
p
I don't know how to fix those
n
Ah, ok, thanks for the help anyway
I wish I could just go back in time
p
try to fiddle around with the longhorn.io.instancemanager objects
You can also try to restore from etcd snapshot but im not sure that it would work
n
yeah Im worried about restoring and borking things even more
75 Views
Open in Slack
PreviousNext
Powered by