adamant-kite-43734
03/13/2024, 2:44 AMcool-architect-86201
03/13/2024, 2:54 AMcool-architect-86201
03/13/2024, 7:55 AMicy-agency-38675
03/13/2024, 8:54 AMicy-agency-38675
03/13/2024, 8:55 AMfaint-sunset-36608
03/13/2024, 8:01 PMsnapshot-cleanup
recurring job either via UI or manifest by following the instructions at https://longhorn.io/docs/1.6.0/snapshots-and-backups/scheduling-backups-and-snapshots/. This will automate the cleanup of system snapshots. You can also try manually cleaning up the snapshots to test how much space you can expect to reclaim. The data from the deleted parent snapshot is coalesced into the child snapshot. One caveat is that it is not possible for the automated cleanup to delete the last snapshot before the volume head. (This is typically where most of the volume's data is stored unless your workload does a lot of writing between snapshots or expansions.) If you feel this behavior is tying up too much space, you can also try creating a recurring snapshot
job with retain == 1
. Every time this job runs, it will create a new snapshot and purge unnecessary ones (creating the new snapshot will make the expand snapshots unnecessary).cool-architect-86201
03/14/2024, 3:25 PMfaint-sunset-36608
03/14/2024, 10:25 PMIf I have a PVC of 1000GB that needs to be resized to 1200GB, do I need to have 2200GB of capacity to ensure creating a snapshot? Because it was just mentioned that resizing by default will create a snapshot.No. The expansion operation does not consume any additional disk space. After the operation, you will have a snapshot with a NOMINAL size of 1000 GB and a volume-head with a NOMINAL size of 1200GB. However, the actual space on disk of the snapshot will be a maximum of 1200 GB and the actual space on disk of the volume-head will be 0 GB. Over time, every new write will increase the actual space on disk of the volume-head. Eventually, you could reach a situation in which the snapshot has an actual size of up to 1000 GB and the volume-head has an actual size of up to 1200 GB. If you reach this point, you really will be using the 2200 GB you suggested above. The goal is to manage space in such a way that there is only ever one snapshot with the majority of a volume's data (up to 1000 GB) and a volume-head with very little additional data. This will keep your actual space on disk close to the nominal size.
Additionally, I'd like to ask, for the cleanup tasks I've set up in the job, should I specify them in each PVC? Is there a solution to make it clean up by default without having to set it for each volume one by one?Volumes that are not otherwise configured belong to the default group for recurring jobs. Just ensure you select "Add to default group" when creating the job to get all volumes. https://longhorn.io/docs/1.6.0/snapshots-and-backups/scheduling-backups-and-snapshots/#add-recurring-jobs-to-the-default-group
cool-architect-86201
03/15/2024, 6:47 AMsnapshot clean
is 0, and I'm unable to modify it. Is "Snapshot Delete" the same as "Snapshot Cleanup"? It's just that the retention for "Snapshot Clean" is 0.
Furthermore, when referring to concurrency here, does it mean how many volumes will undergo such operations simultaneously? Is that correct?
whether it's adding or deleting snapshots, it all happens automatically in the background without affecting the current use of the pod?cool-architect-86201
03/15/2024, 7:31 AMfaint-sunset-36608
03/15/2024, 1:04 PMBecause sparse files can be confusing for many people, and it takes some time to merge snapshots or delete snapshots.• Sparse files allow for overprovisioning. If we used regular files, a 1000 GB Longhorn volume would require and consume 1000 GB of disk space, even when empty. You would only be able to provision a single 1000 GB volume 1000 GB disk, leaving a whole bunch of physical disk space unused. On the one hand, that would make the space usage very easy to understand. On the other hand, it might be pretty inefficient. • Sparse files enabled our snapshot chain. Snapshots are a key feature of Longhorn. When Longhorn reads data from a volume, it first checks volume-head for that data, then the previous snapshot, then the previous snapshot, etc. By using sparse files, we can look to holes in the "current" snapshot to indicate that no data has been written at a location and we should look further back. Regular files will always have SOMETHING at every location (even if it is just zeros), so Longhorn wouldn't know whether to look further back in the chain.
There's a phenomenon I'm curious about: ActualSize > Size. It seems we've already "oversold" because my PVC applied for 750GB of space, but it's actually using close to 800GB, which surprises me. Is there a BUG?I don't see anything unexpected here, unless I'm missing something. • You had a 700 GB volume and wrote 620 GB to it. • You (or the system during a replica rebuild) created a snapshot. • You wrote another 123 GB to the volume. Longhorn kept all of these 123 GB on the NEW volume head (probably volume-head-001), leaving the snapshot consistent. You could roll back to it at any time. • You expanded the volume to 750 GB, creating a snapshot in the process. • You wrote another 56 GB to the volume. Longhorn kept all of these 56 GB on the NEW volume head (volume-head-002), leaving the snapshot consistent. You could roll back to it at any time. If you (for example) delete the 620 GB snapshot, it will coalesce into the 123 GB snapshot. Probably most of the 123 GB of writes are in the same locations as the previous 620 GB of writes, so you will "gain" back ~123 GB by doing that.
faint-sunset-36608
03/15/2024, 1:17 PMFurthermore, when referring to concurrency here, does it mean how many volumes will undergo such operations simultaneously? Is that correct?Correct.
Additionally, the default retention forThe delete job will delete all kinds of snapshots (including intentionally created ones). It is probably fine in your use case, since it sounds like you don't want any snapshots at all.is 0, and I'm unable to modify it. Is "Snapshot Delete" the same as "Snapshot Cleanup"? It's just that the retention for "Snapshot Clean" is 0.snapshot clean
retention == 0
is what you want if I understand correctly. It is only zero for the clean job because clean only deletes unintentional / system-generated snapshots. There is probably no reason you would want to retain any of them. On the other hand, the delete job might delete intentional snapshots. You can choose how many of these you want to stick around.cool-architect-86201
03/16/2024, 3:13 AMfaint-sunset-36608
03/19/2024, 5:41 PMFor example, if I expand from 1000GB to 1200GB, my disk must actually have more than 1200GB of space. Otherwise, due to concurrent issues, data may be written during snapshot creation.Is this understanding correct?Yes. Writes issued between the moment the snapshot is taken for the expansion and the moment you are able to delete the snapshot MAY cause the actual space to exceed 1200 GB. This is more likely if your volume is mostly full. (For example, if you have actually written 1000 GB of data to the volume before the expansion, then you write 200 GB of data to the new blocks from the expansion, and ALSO write some data to the previously existing blocks, your volume will exceed 1200 GB in actual size.
Actually, during yesterday's snapshot deletion process, I found it to be slow. To be honest, this concerns me a bit. I must reserve some extra space so that it can smoothly expand when needed. How can I speed up the time it takes to delete and merge snapshots?I understand your concern, but there isn't a particularly good way to speed up the process, unfortunately. The operation involves moving chunks of data from one sparse file to another. With sizes this large, that can take an appreciable amount of time.