Hi folks I ve been using rancher desktop for awhile now and rancher-users #rancher-desktop

Title

numerous-motorcycle-43166

04/26/2023, 2:56 PM

Hi folks, I've been using rancher-desktop for awhile now and it's great! I've recently started using it with

skaffold

and it works most of the time, however after 2-3 reloads from my code changes, the node becomes unusable due to disk pressure. I'm using linux with rancher desktop 1.8.1. Has anyone ran into this before or have any suggestions to try to help avoid this disk pressure issue?

The only way I've found to fix it so far is to completely reset my k8s environment in rancher-desktop and rebuild everything, which takes 30 min or so 😕

fast-garage-66093

04/27/2023, 3:32 AM

The volume of the VM that stores images has a maximum size of 100GiB by default. You can stop Rancher Desktop, increase the size, and restart Rancher Desktop, and it will grow the partition to use the additional space. I've shown how to do this on macOS at https://rancher-users.slack.com/archives/C0200L1N1MM/p1679346547592089?thread_ts=1674224343.892149&cid=C0200L1N1MM

I don't have a Linux system here right now to test, so I'm not sure if this is correct, but it should be something like

/opt/rancher-desktop/resources/resources/linux/lima/bin/qemu-img resize ~/.local/share/rancher-desktop/lima/0/diffdisk +10G

You need to double-check that I guessed the right paths, and you may want to add more than just 10GB if you regularly run out of space with 100GB.

Make sure Rancher Desktop is stopped when you resize the volume, and never try to decrease the size; it will break the filesystem.

I also don't know what mechanisms

skaffold

has to delete images no longer needed, but it sounds like you should frequently prune the image store.

I think

kubelet

is going to do some garbage collection when a threshold is reached, but maybe that threshold is too low, so you exhaust the disk before GC can kick in.

numerous-motorcycle-43166

04/27/2023, 12:59 PM

Thanks, I'll give that a try. When I run

docker system prune

it does show 90+ GB. I've been running skaffold with

--no-prune=false --cache-artifacts=false

which should prune the images more often; so hopefully that helps too

fast-garage-66093

04/27/2023, 4:25 PM

Please let us all know how it works out (both increasing the volume size, and pruning images)

numerous-motorcycle-43166

04/27/2023, 5:37 PM

the pruning by itself doesn't seem to do the trick so I'll try resizing. I don't seem to have

qemu-img

in that path though, only `limactl`:

❯ ll /opt/rancher-desktop/resources/resources/linux/lima/bin/            
total 19844
drwxr-xr-x 2 root root     4096 Apr 19 12:57 ./
drwxr-xr-x 4 root root     4096 Apr 19 12:57 ../
-rwxr-xr-x 1 root root 20312064 Feb 28 16:52 limactl*

fast-garage-66093

04/27/2023, 5:39 PM

Oh, we don't ship qemu with Rancher Desktop on Linux; it is a dependencies that will have been installed by your package manager. You should have

qemu-img

on the

PATH

✅ 1

numerous-motorcycle-43166

04/27/2023, 5:40 PM

👍 thanks,

qemu-img resize ~/.local/share/rancher-desktop/lima/0/diffdisk +10G

worked. Will test the results of a bigger size now.

fast-garage-66093

04/27/2023, 5:42 PM

I don't know how much free space you have on your host, but adding 10G may not be enough if you already bump into 100G after 2-3 builds, and pruning doesn't help.

numerous-motorcycle-43166

04/27/2023, 5:49 PM

693G free space 😄 you're right, probably need to increase some more

do you know if I need to re-run the resize command after resetting k8s through the UI?

fast-garage-66093

04/27/2023, 5:54 PM

No, the only time you will need to redo this is after a factory-reset because that deletes the volume

👍 1

numerous-motorcycle-43166

05/02/2023, 3:45 PM

little update here, 200G and using the

--no-prune=false --cache-artifacts=false

options with skaffold has been pretty reliable

👍 1

#rancher-desktop Join Slack