This message was deleted.

Since our business scenario frequently involves disk expansion, is it possible to disable snapshots and directly expand the disk? Because our business data consists of blockchain data, which does not require snapshots. Otherwise, this would greatly waste our storage space.

Hello I’m out of office this week. For the actual size, you can see https://longhorn.io/docs/1.6.0/nodes-and-volumes/volumes/volume-size/

Hello @cool-architect-86201. Longhorn must take a snapshot to expand the disk. However, snapshots can be automatically cleaned up to allow for space reclamation. Try creating a

snapshot-cleanup

recurring job either via UI or manifest by following the instructions at https://longhorn.io/docs/1.6.0/snapshots-and-backups/scheduling-backups-and-snapshots/. This will automate the cleanup of system snapshots. You can also try manually cleaning up the snapshots to test how much space you can expect to reclaim. The data from the deleted parent snapshot is coalesced into the child snapshot. One caveat is that it is not possible for the automated cleanup to delete the last snapshot before the volume head. (This is typically where most of the volume's data is stored unless your workload does a lot of writing between snapshots or expansions.) If you feel this behavior is tying up too much space, you can also try creating a recurring

snapshot

job with

retain == 1

. Every time this job runs, it will create a new snapshot and purge unnecessary ones (creating the new snapshot will make the expand snapshots unnecessary).

Okay, thank you. @faint-sunset-36608 If I have a PVC of 1000GB that needs to be resized to 1200GB, do I need to have 2200GB of capacity to ensure creating a snapshot? Because it was just mentioned that resizing by default will create a snapshot. Additionally, I'd like to ask, for the cleanup tasks I've set up in the job, should I specify them in each PVC? Is there a solution to make it clean up by default without having to set it for each volume one by one?

If I have a PVC of 1000GB that needs to be resized to 1200GB, do I need to have 2200GB of capacity to ensure creating a snapshot? Because it was just mentioned that resizing by default will create a snapshot.

No. The expansion operation does not consume any additional disk space. After the operation, you will have a snapshot with a NOMINAL size of 1000 GB and a volume-head with a NOMINAL size of 1200GB. However, the actual space on disk of the snapshot will be a maximum of 1200 GB and the actual space on disk of the volume-head will be 0 GB. Over time, every new write will increase the actual space on disk of the volume-head. Eventually, you could reach a situation in which the snapshot has an actual size of up to 1000 GB and the volume-head has an actual size of up to 1200 GB. If you reach this point, you really will be using the 2200 GB you suggested above. The goal is to manage space in such a way that there is only ever one snapshot with the majority of a volume's data (up to 1000 GB) and a volume-head with very little additional data. This will keep your actual space on disk close to the nominal size.

Additionally, I'd like to ask, for the cleanup tasks I've set up in the job, should I specify them in each PVC? Is there a solution to make it clean up by default without having to set it for each volume one by one?

Volumes that are not otherwise configured belong to the default group for recurring jobs. Just ensure you select "Add to default group" when creating the job to get all volumes. https://longhorn.io/docs/1.6.0/snapshots-and-backups/scheduling-backups-and-snapshots/#add-recurring-jobs-to-the-default-group

Thank you for patiently answering my questions👍. I've learned a lot from it. It seems that sparse files are being used here. I've looked up some information, but I don't quite understand the benefits of doing this. Because sparse files can be confusing for many people, and it takes some time to merge snapshots or delete snapshots. Additionally, the default retention for

snapshot clean

is 0, and I'm unable to modify it. Is "Snapshot Delete" the same as "Snapshot Cleanup"? It's just that the retention for "Snapshot Clean" is 0. Furthermore, when referring to concurrency here, does it mean how many volumes will undergo such operations simultaneously? Is that correct? whether it's adding or deleting snapshots, it all happens automatically in the background without affecting the current use of the pod?

Because sparse files can be confusing for many people, and it takes some time to merge snapshots or delete snapshots.

• Sparse files allow for overprovisioning. If we used regular files, a 1000 GB Longhorn volume would require and consume 1000 GB of disk space, even when empty. You would only be able to provision a single 1000 GB volume 1000 GB disk, leaving a whole bunch of physical disk space unused. On the one hand, that would make the space usage very easy to understand. On the other hand, it might be pretty inefficient. • Sparse files enabled our snapshot chain. Snapshots are a key feature of Longhorn. When Longhorn reads data from a volume, it first checks volume-head for that data, then the previous snapshot, then the previous snapshot, etc. By using sparse files, we can look to holes in the "current" snapshot to indicate that no data has been written at a location and we should look further back. Regular files will always have SOMETHING at every location (even if it is just zeros), so Longhorn wouldn't know whether to look further back in the chain.

There's a phenomenon I'm curious about: ActualSize > Size. It seems we've already "oversold" because my PVC applied for 750GB of space, but it's actually using close to 800GB, which surprises me. Is there a BUG?

I don't see anything unexpected here, unless I'm missing something. • You had a 700 GB volume and wrote 620 GB to it. • You (or the system during a replica rebuild) created a snapshot. • You wrote another 123 GB to the volume. Longhorn kept all of these 123 GB on the NEW volume head (probably volume-head-001), leaving the snapshot consistent. You could roll back to it at any time. • You expanded the volume to 750 GB, creating a snapshot in the process. • You wrote another 56 GB to the volume. Longhorn kept all of these 56 GB on the NEW volume head (volume-head-002), leaving the snapshot consistent. You could roll back to it at any time. If you (for example) delete the 620 GB snapshot, it will coalesce into the 123 GB snapshot. Probably most of the 123 GB of writes are in the same locations as the previous 620 GB of writes, so you will "gain" back ~123 GB by doing that.

I'm amazed👍. To ensure my understanding is correct, let me summarize again. The key point is that the snapshot creation and pod (program) writing processes are concurrent behaviors because our business (blockchain) is still syncing data. Longhorn must ensure data integrity, so some data is redundantly written between the two volumes. This results in the actual disk usage exceeding the PVC set value. For example, if I expand from 1000GB to 1200GB, my disk must actually have more than 1200GB of space. Otherwise, due to concurrent issues, data may be written during snapshot creation.Is this understanding correct? Actually, during yesterday's snapshot deletion process, I found it to be slow. To be honest, this concerns me a bit. I must reserve some extra space so that it can smoothly expand when needed. How can I speed up the time it takes to delete and merge snapshots? Also, I want to confirm: if the pod has stopped and the Volume is in a detached state, can snapshot deletion and merging operations be performed? Because in this case, there is no concurrent writing process. SUSE's engineering culture is strong, and I really like it.

For example, if I expand from 1000GB to 1200GB, my disk must actually have more than 1200GB of space. Otherwise, due to concurrent issues, data may be written during snapshot creation.Is this understanding correct?

Yes. Writes issued between the moment the snapshot is taken for the expansion and the moment you are able to delete the snapshot MAY cause the actual space to exceed 1200 GB. This is more likely if your volume is mostly full. (For example, if you have actually written 1000 GB of data to the volume before the expansion, then you write 200 GB of data to the new blocks from the expansion, and ALSO write some data to the previously existing blocks, your volume will exceed 1200 GB in actual size.

Actually, during yesterday's snapshot deletion process, I found it to be slow. To be honest, this concerns me a bit. I must reserve some extra space so that it can smoothly expand when needed. How can I speed up the time it takes to delete and merge snapshots?

I understand your concern, but there isn't a particularly good way to speed up the process, unfortunately. The operation involves moving chunks of data from one sparse file to another. With sizes this large, that can take an appreciable amount of time.