This message was deleted.
# longhorn-storagea
adamant-kite-43734
01/05/2023, 1:08 AMThis message was deleted.
b
billowy-painting-56466
01/05/2023, 2:28 AMWhat is your Longhorn version?
Can you provide the manger log?
cc @famous-shampoo-18483
f
famous-shampoo-18483
01/05/2023, 2:29 AMAnd what’s your backing image size?
q
quaint-alarm-7893
01/05/2023, 2:29 AM1.3.2. it's running with harvester. 1.1.2
as for the log, how can i pull it?
quaint-alarm-7893
01/05/2023, 2:29 AMit's 100gb
quaint-alarm-7893
01/05/2023, 2:29 AMactual size is like 40gb
quaint-alarm-7893
01/05/2023, 2:30 AMcheck-that, 28gb
f
famous-shampoo-18483
01/05/2023, 2:30 AMOK. I guess you are uploading the file as a backing image. There is a bug for the Longhorn UI: https://github.com/longhorn/longhorn/issues/4902
q
quaint-alarm-7893
01/05/2023, 2:30 AMi ran: k logs longhorn-manager-9d78v -n longhorn-system earlier, and it shows the logs, but it seems like it reset swhenver ti resets the pod
quaint-alarm-7893
01/05/2023, 2:31 AMi'm not uploading, it's an export from a volume.
quaint-alarm-7893
01/05/2023, 2:31 AMharvester generates something like this to create it:
Copy code
apiVersion: <http://longhorn.io/v1beta2|longhorn.io/v1beta2>
kind: BackingImage
metadata:
annotations:
<http://harvesterhci.io/imageId|harvesterhci.io/imageId>: default/image-fs65q
name: default-image-fs65q
namespace: longhorn-system
spec:
sourceParameters:
export-type: raw
volume-name: pvc-7ba4e01b-8be4-4a3c-9f43-1564eaac9027
sourceType: export-from-volumequaint-alarm-7893
01/05/2023, 2:32 AMit's similar to this issue:
https://rancher-users.slack.com/archives/CC2UQM49Y/p1671422861524339
b
billowy-painting-56466
01/05/2023, 2:35 AMi ran: k logs longhorn-manager-9d78v -n longhorn-system earlier, and it shows the logs, but it seems like it reset swhenver ti resets the podTry
k logs longhorn-manager-9d78v -n longhorn-system --previousf
famous-shampoo-18483
01/05/2023, 2:36 AMFor the backing image part, you can check the log of pod
backing-image-ds-<backing image name>longhorn-systemq
quaint-alarm-7893
01/05/2023, 2:44 AMokay, give me a few... running it now so i can get fresh/clean logs
quaint-alarm-7893
01/05/2023, 2:52 AMlogs attached
quaint-alarm-7893
01/05/2023, 2:52 AMi streamed the log for the backing image
quaint-alarm-7893
01/05/2023, 2:53 AMbut because the manager changes depending on what node it fails with, (it seems to do it with all of them), i had to grab it with the --previous option you gave me. (that's nice to know btw)
f
famous-shampoo-18483
01/05/2023, 2:54 AMThe error log in the backing-image-ds pod:
famous-shampoo-18483
01/05/2023, 2:56 AMIt seems that somehow the HTTP connection between this pod and the replica process of the running volume is closed.
q
quaint-alarm-7893
01/05/2023, 2:56 AMthe line prior to that is the last line before the pod terminates. i check it with --previous as well
quaint-alarm-7893
01/05/2023, 2:57 AMany ideas on how to troubleshoot why? should i just down a bunch of vms see if it's a load issue or something?
f
famous-shampoo-18483
01/05/2023, 3:03 AM1. Make sure there is no network issue or other kind of resource exhaustion issues.
2. Need to figure out which replica of the volume is responsible for exporting the data to the backing image. Then check the log. The replica processes are in instance-manager-r pods hence you need to check the pod logs.
b
billowy-painting-56466
01/05/2023, 3:06 AMI've created and issue for this: https://github.com/longhorn/longhorn/issues/5209
q
quaint-alarm-7893
01/05/2023, 3:09 AMokay, i'll mess around with it, and keep an eye on the issue. thanks!
f
famous-shampoo-18483
01/05/2023, 3:12 AMBTW, you can directly generate the support bundle and upload it in this GitHub ticket.
đź‘Ť 1
4 Views