https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
m
Copy code
2026-02-23T23:29:53.907631532Z time="2026-02-23T23:29:53Z" level=info msg="CreateSnapshot is called with req source_volume_id:\"pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f\"  name:\"snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b\"" func="csi.(*ControllerServer).CreateSnapshot" file="controller_server.go:652" component=csi-controller-server function=CreateSnapshot
2026-02-23T23:29:53.915888798Z time="2026-02-23T23:29:53Z" level=info msg="CreateSnapshot: rsp: {\"snapshot\":{\"creation_time\":{\"seconds\":1771889126},\"size_bytes\":128849018880,\"snapshot_id\":\"<bak://pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/backup-f3c8ac3bc6584cd7>\",\"source_volume_id\":\"pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f\"}}" func=csi.logGRPC file="server.go:139"
2026-02-23T23:34:09.946647305Z time="2026-02-23T23:34:09Z" level=info msg="GetPluginInfo: req: {}" func=csi.logGRPC file="server.go:130"
2026-02-23T23:34:09.946700534Z time="2026-02-23T23:34:09Z" level=info msg="GetPluginInfo: rsp: {\"name\":\"<http://driver.longhorn.io|driver.longhorn.io>\",\"vendor_version\":\"v1.9.2\"}" func=csi.logGRPC file="server.go:139"
2026-02-23T23:34:09.947466365Z time="2026-02-23T23:34:09Z" level=info msg="CreateSnapshot: req: {\"name\":\"snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b\",\"source_volume_id\":\"pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f\"}" func=csi.logGRPC file="server.go:130"
2026-02-23T23:34:09.947518121Z time="2026-02-23T23:34:09Z" level=info msg="CreateSnapshot is called with req source_volume_id:\"pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f\"  name:\"snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b\"" func="csi.(*ControllerServer).CreateSnapshot" file="controller_server.go:652" component=csi-controller-server function=CreateSnapshot
2026-02-23T23:34:09.952341227Z time="2026-02-23T23:34:09Z" level=error msg="Failed to create snapshot" func="csi.(*ControllerServer).CreateSnapshot.func1" file="controller_server.go:658" component=csi-controller-server error="rpc error: code = Internal desc = failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error" function=CreateSnapshot
2026-02-23T23:34:09.952414393Z time="2026-02-23T23:34:09Z" level=error msg="CreateSnapshot: err: rpc error: code = Internal desc = failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error" func=csi.logGRPC file="server.go:136"
Copy code
2026-02-23T23:30:50.733687109Z time="2026-02-23T23:30:50Z" level=error msg="Failed to perform backup for volume pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f snapshot snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b" func=backupstore.CreateDeltaBlockBackup.func4 file="deltablock.go:310" backup_type=full error="failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error"
2026-02-23T23:30:50.736726821Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:92" filepath=backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/locks/lock-ced4682399b945e3.lck kind=nfs object=config pkg=backupstore reason=complete
2026-02-23T23:30:50.737076133Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=info msg="Stored lock backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/locks/lock-ced4682399b945e3.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:204" pkg=backupstore
2026-02-23T23:30:50.737345185Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:81" filepath=backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/locks/lock-ced4682399b945e3.lck kind=nfs object=config pkg=backupstore reason=start
2026-02-23T23:30:50.758350105Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:92" filepath=backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/locks/lock-ced4682399b945e3.lck kind=nfs object=config pkg=backupstore reason=complete
2026-02-23T23:30:50.758838286Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=warning msg="backup of volume [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f] and snapshot [volume-snap-snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b.img] already completed or failed, skip the status update" func="replica.(*BackupStatus).UpdateBackupStatus" file="backup.go:61"
2026-02-23T23:30:50.758851891Z time="2026-02-23T23:30:50Z" level=info msg="Stored lock backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/locks/lock-ced4682399b945e3.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:204" pkg=backupstore
2026-02-23T23:30:50.766173663Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=info msg="Removed lock backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/locks/lock-ced4682399b945e3.lck type 1 on backupstore" func=backupstore.removeLock file="lock.go:194" pkg=backupstore
2026-02-23T23:30:50.803190809Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=warning msg="backup of volume [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f] and snapshot [volume-snap-snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b.img] already completed or failed, skip the status update" func="replica.(*BackupStatus).UpdateBackupStatus" file="backup.go:61"
2026-02-23T23:30:50.803512679Z [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f-r-9b0c2fb0] time="2026-02-23T23:30:50Z" level=error msg="Failed to read volume pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f snapshot snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b block at offset 74740400128 size 2097152" func=backupstore.backupMapping file="deltablock.go:489" error="volume [pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f] and snapshot [volume-snap-snapshot-92b2d590-eb3c-49bf-b603-66ff777e8e4b.img] are not opened"
Copy code
2026-02-23T23:34:09.926442768Z I0223 23:34:09.926277       1 snapshot_controller.go:339] createSnapshotWrapper: Creating snapshot for content snapcontent-92b2d590-eb3c-49bf-b603-66ff777e8e4b through the plugin ...
2026-02-23T23:34:09.952636976Z I0223 23:34:09.952557       1 snapshot_controller.go:373] createSnapshotWrapper: CreateSnapshot for content snapcontent-92b2d590-eb3c-49bf-b603-66ff777e8e4b returned error: rpc error: code = Internal desc = failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error
2026-02-23T23:34:09.964996572Z E0223 23:34:09.964905       1 snapshot_controller.go:148] checkandUpdateContentStatus [snapcontent-92b2d590-eb3c-49bf-b603-66ff777e8e4b]: error occurred failed to take snapshot of the volume pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f: "rpc error: code = Internal desc = failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error"
2026-02-23T23:34:09.965032578Z E0223 23:34:09.964956       1 snapshot_controller_base.go:279] could not sync content "snapcontent-92b2d590-eb3c-49bf-b603-66ff777e8e4b": failed to take snapshot of the volume pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f: "rpc error: code = Internal desc = failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error"
2026-02-23T23:34:09.965131742Z I0223 23:34:09.965031       1 event.go:377] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-92b2d590-eb3c-49bf-b603-66ff777e8e4b", UID:"03f64b8a-05fa-4392-83b8-accd3f946720", APIVersion:"<http://snapshot.storage.k8s.io/v1|snapshot.storage.k8s.io/v1>", ResourceVersion:"273183362", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCheckandUpdateFailed' Failed to check and update snapshot content: failed to take snapshot of the volume pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f: "rpc error: code = Internal desc = failed to write data during saving blocks: close /var/lib/longhorn-backupstore-mounts/10_10_10_3/volume1/jenkins/harvester-backups/backupstore/volumes/2d/79/pvc-0c0d9cb9-7e9a-4be7-8994-2111d0f58f0f/blocks/07/08/0708fb0244899494bfcce013943913fc2889bbf26ad31f368fb13586a2fc00ba.blk.tmp.1771889360468586898: input/output error"
f
Hey there Doug, what is the harvester version you are using ?
Also a full support bundle would be helpful in that case, the log excerpt might not lead to the root cause
m
We are on v1.6.1
and here's the support bundle.
supportbundle.zip
f
Seems like almost all nodes report timeouts related with the NFS on 10.10.10.3 but I can't see there so you can point to other NFS or try diagnosing this one behind the IP if any outages happened and restart the backup creation
m
do you know which interface it's failing from?
22 Views
Open in Slack
PreviousNext
Powered by