powerful-easter-15334
09/25/2025, 5:31 AMpowerful-easter-15334
09/25/2025, 5:32 AM[11467358.262783] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-f9d2ff8b835dbf910c5f761bd7bdae95ce7685efbd07876b95556d1ea16d78ed.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6998669_cc33_4fd5_a34d_0368396d3afe.slice/cri-containerd-f9d2ff8b835dbf910c5f761bd7bdae95ce7685efbd07876b95556d1ea16d78ed.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6998669_cc33_4fd5_a34d_0368396d3afe.slice/cri-containerd-f9d2ff8b835dbf910c5f761bd7bdae95ce7685efbd07876b95556d1ea16d78ed.scope,task=qemu-system-x86,pid=10476,uid=107
[11467358.262870] Memory cgroup out of memory: Killed process 10476 (qemu-system-x86) total-vm:51480644kB, anon-rss:49988500kB, file-rss:5532kB, shmem-rss:4kB, UID:107 pgtables:98416kB oom_score_adj:897
powerful-easter-15334
09/25/2025, 5:32 AMpowerful-easter-15334
09/25/2025, 5:33 AM<http://harvesterhci.io/reservedMemory|harvesterhci.io/reservedMemory>: 512Mi
I added this. Given the huge amount of RAM (it's an Oracle db), should I increase this value? The docs only mention 512 if 256 still causes issuespowerful-easter-15334
09/25/2025, 5:35 AMpowerful-easter-15334
09/25/2025, 5:37 AMpowerful-easter-15334
09/25/2025, 5:37 AMpowerful-easter-15334
09/25/2025, 5:52 AMA VM that is configured to have 1 CPU, 64 Gi Memory, 1 Volume and 1 NIC will get around 250 Mi Memory Overhead when the ratio is "1.0". The VM memory size does not have a big influence on the computing of Memory Overhead. The overhead of guest OS pagetables needs one bit for every 512b of RAM size.
Yet VM memory doesn't have a big influence - what does? CPU? Number and size of attached volumes? IO activity?
My hypothesis is that it's still a reservedMemory thing, but I'll wait for a second opinionenough-australia-5601
09/25/2025, 8:14 AMpowerful-easter-15334
09/25/2025, 8:15 AMpowerful-easter-15334
09/25/2025, 8:15 AMenough-australia-5601
09/25/2025, 10:30 AMpowerful-easter-15334
09/25/2025, 10:31 AMpowerful-easter-15334
09/25/2025, 10:32 AMpowerful-easter-15334
09/25/2025, 11:08 AMbumpy-portugal-40754
09/25/2025, 1:40 PMpowerful-easter-15334
09/25/2025, 1:41 PMpowerful-easter-15334
09/25/2025, 1:42 PMbumpy-portugal-40754
09/25/2025, 1:43 PMpowerful-easter-15334
09/25/2025, 1:44 PMpowerful-easter-15334
09/25/2025, 1:44 PMbumpy-portugal-40754
09/25/2025, 1:47 PMenough-australia-5601
09/25/2025, 2:46 PMOracle DB which is allergic to uptimeThis is gold, thanks for the laugh π€£ As I understand it, virtual machines will not consume memory unless it is written to. This means that allocating 48Gi of memory to a VM does not immediately allocate all of the 48Gi of memory, but rather in chunks over time, as the workload within the VM dirties the memory pages. The hypervisor can easily determine if a memory page has never been written to. Therefore on startup the hypervisor (i.e. the qemu process) only needs as much memory as is actually being used in the VM, since the clean memory pages don't yet need to be allocated. Over time the hypervisor will then increase it's memory footprint as more and more memory pages in the VM get dirtied. However the hypervisor can not (by itself) see when memory pages in the VM are freed up again. Therefore its memory footprint grows until it holds all of the VM's memory - plus its own overhead of course. This means that a VM with a workload that doesn't fully utilize the entire memory capacity of the VM will probably never see OOMs. But a VM with the same resources allocated may see OOMs if the workload in the VM churns memory a lot[1]. The remedy to that is memory ballooning, where a special virtual device is installed in the VM and the guest OS has a special driver for this device. The memory ballooning driver will allocate free memory inside the virtual machine and communicate via the virtual ballooning device to the hypervisor that this memory is free and it's ok to release it. Otherwise the hypervisor can't ever shrink the actual memory footprint of a VM. Qemu and Libvirt support memory ballooning, however to my knowledge KubeVirt and Harvester don't. Therefore it's necessary that the virt-launcher Pod, which contains the qemu process that is the VM, has sufficient memory resources assigned to cover the entirety of the VM itself plus the overhead, because otherwise it will eventually run out of memory. Now here's the problem: It's pretty hard to determine what the overhead is as well, because this depends on the VM's configuration. It's quite reasonable to assume that more virtual devices require more memory overhead of the hypervisor, but how much exactly is not obvious. I'd assume that it doesn't take long at all until all 48Gi of memory of your VM have been dirtied, if you're doing lots of IO. Once that's the case, the qemu process will hold 48Gi of memory. Then only the reserved memory remains for the hypervisor process and its helpers itself (i.e. for qemu, virt-launcher...). Note that the reserved memory setting is only applied if the
additional-guest-memory-overhead-ratio
is 0
, otherwise the overhead-ratio will be used.
[1]: Lots of disk IO usually means lots of memory churn as well, because the guest OS kernel will utilize that memory for filesystem caching.
Here is an example from my workstation:
β ~ ββΊ free -m
total used free shared buff/cache available
Mem: 63987 12854 6338 1695 47207 51133
You can see the free
indicates that only ~12Gi are actually allocated, however including buffers and caches there are almost 60Gi "in use". Now my OS could free up those buffers and caches if required, hence there are still 51Gi "available". But since all of that memory is dirty pages, if this was a virtual machine, the hypervisor would need to hold all of that in memory, since it can't tell which pages belong to the 12Gi in use and which are just cache.bumpy-portugal-40754
09/25/2025, 2:52 PMpowerful-easter-15334
09/25/2025, 2:54 PMpowerful-easter-15334
09/25/2025, 2:54 PMpowerful-easter-15334
09/25/2025, 2:56 PMpowerful-easter-15334
09/25/2025, 2:57 PMbumpy-portugal-40754
09/25/2025, 3:01 PMpowerful-easter-15334
09/25/2025, 3:01 PMpowerful-easter-15334
09/25/2025, 3:03 PMbumpy-portugal-40754
09/25/2025, 3:09 PMpowerful-easter-15334
09/25/2025, 3:10 PMbumpy-portugal-40754
09/25/2025, 3:12 PMpowerful-easter-15334
09/25/2025, 3:13 PMenough-australia-5601
09/25/2025, 4:34 PMThis is not about memory used INSIDE the VM! It's about the memory outside of the VM, but inside the pod for virt-launcher, qemu, buffers etc.
You
can be lucky if the VM is not using all assigned memory, and don't see
OOM kills for a long time. But if the VM is starting to use their
configured memory almost completely, then the additional overhead is
saving your (VM) live, or running straight into OOM kill.The memory inside the VM influences the memory consumption of the hypervisor, because if a memory page is never dirtied, the hypervisor never needs to allocate it. But if a memory page is dirtied only once, the hypervisor needs to allocate it until the VM is restarted (unless something like memory ballooning is used). Therefore memory behavior inside the VM influences indirectly memory behavior of the Pod and therefore has an influence on whether or not you see OOMs when there is too little overhead accounted for. The cumulative memory consumption inside the VM is a lower bound for the memory requirement of the Pod
Ask Suse why there are 2 ways for doing the same thing.Because it's not the same thing. When you define a VM in Harvester, you give one number for how much memory it should have. If you consider that number to include the hypervisor overhead, then the VM doesn't get as much memory as you assigned. Some people consider this to be unacceptable. Conversely you consider that number to be the guaranteed amount of memory that the VM is going to get, i.e. to exclude the hypervisor overhead, then the hypervisor overhead is going to push the total memory consumption of that VM above the specified number. Some people consider this to be unacceptable as well. By giving two options, you have the choice. Do you care about never exceeding the amount of memory you specify for a VM, or do you care about guaranteeing the amount of memory you specify. It's up to you to look at your use case and figure out which one is better for you.
bumpy-portugal-40754
09/25/2025, 5:02 PM