https://rancher.com/ logo
Title
n

numerous-motorcycle-43166

04/26/2023, 2:56 PM
Hi folks, I've been using rancher-desktop for awhile now and it's great! I've recently started using it with
skaffold
and it works most of the time, however after 2-3 reloads from my code changes, the node becomes unusable due to disk pressure. I'm using linux with rancher desktop 1.8.1. Has anyone ran into this before or have any suggestions to try to help avoid this disk pressure issue?
The only way I've found to fix it so far is to completely reset my k8s environment in rancher-desktop and rebuild everything, which takes 30 min or so 😕
f

fast-garage-66093

04/27/2023, 3:32 AM
The volume of the VM that stores images has a maximum size of 100GiB by default. You can stop Rancher Desktop, increase the size, and restart Rancher Desktop, and it will grow the partition to use the additional space. I've shown how to do this on macOS at https://rancher-users.slack.com/archives/C0200L1N1MM/p1679346547592089?thread_ts=1674224343.892149&cid=C0200L1N1MM
I don't have a Linux system here right now to test, so I'm not sure if this is correct, but it should be something like
/opt/rancher-desktop/resources/resources/linux/lima/bin/qemu-img resize ~/.local/share/rancher-desktop/lima/0/diffdisk +10G
You need to double-check that I guessed the right paths, and you may want to add more than just 10GB if you regularly run out of space with 100GB.
Make sure Rancher Desktop is stopped when you resize the volume, and never try to decrease the size; it will break the filesystem.
I also don't know what mechanisms
skaffold
has to delete images no longer needed, but it sounds like you should frequently prune the image store.
I think
kubelet
is going to do some garbage collection when a threshold is reached, but maybe that threshold is too low, so you exhaust the disk before GC can kick in.
n

numerous-motorcycle-43166

04/27/2023, 12:59 PM
Thanks, I'll give that a try. When I run
docker system prune
it does show 90+ GB. I've been running skaffold with
--no-prune=false --cache-artifacts=false
which should prune the images more often; so hopefully that helps too
f

fast-garage-66093

04/27/2023, 4:25 PM
Please let us all know how it works out (both increasing the volume size, and pruning images)
n

numerous-motorcycle-43166

04/27/2023, 5:37 PM
the pruning by itself doesn't seem to do the trick so I'll try resizing. I don't seem to have
qemu-img
in that path though, only `limactl`:
❯ ll /opt/rancher-desktop/resources/resources/linux/lima/bin/            
total 19844
drwxr-xr-x 2 root root     4096 Apr 19 12:57 ./
drwxr-xr-x 4 root root     4096 Apr 19 12:57 ../
-rwxr-xr-x 1 root root 20312064 Feb 28 16:52 limactl*
f

fast-garage-66093

04/27/2023, 5:39 PM
Oh, we don't ship qemu with Rancher Desktop on Linux; it is a dependencies that will have been installed by your package manager. You should have
qemu-img
on the
PATH
1
n

numerous-motorcycle-43166

04/27/2023, 5:40 PM
👍 thanks,
qemu-img resize ~/.local/share/rancher-desktop/lima/0/diffdisk +10G
worked. Will test the results of a bigger size now.
f

fast-garage-66093

04/27/2023, 5:42 PM
I don't know how much free space you have on your host, but adding 10G may not be enough if you already bump into 100G after 2-3 builds, and pruning doesn't help.
n

numerous-motorcycle-43166

04/27/2023, 5:49 PM
693G free space 😄 you're right, probably need to increase some more
do you know if I need to re-run the resize command after resetting k8s through the UI?
f

fast-garage-66093

04/27/2023, 5:54 PM
No, the only time you will need to redo this is after a factory-reset because that deletes the volume
👍 1
n

numerous-motorcycle-43166

05/02/2023, 3:45 PM
little update here, 200G and using the
--no-prune=false --cache-artifacts=false
options with skaffold has been pretty reliable
👍 1