https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
p
This is from the Rancher dashboard
This is from a running VM, which was allocated only 32GB RAM
Bottom line being that all running VMs are using up 2x as much RAM as allocated. Is this normal?
b
Reserve isn't the same as requested
Reserve is what is dedicated, requests is what it can use. By default Reserve is 1/2 of whatever you set the vm to in the ui
And there's an overcommit setting for VM density too
So there's two allocations and it sounds or looks like you might be getting them confused based off the screenshots here
p
Ah, alright, I understand the reserve in the case of this particular VM, would be 32/2 (16). I take it then that the overcommit setting would explain the RAM usage being double?
b
Making a 64Gi vm sets a 32Gi reserve. The oom kicks in after 64 Gi of usage. Those numbers change when there's a migration.
p
Yes, that I understand. But here it displays 64GB usage when I specified 32 for the VM.
This is the screen on Grafana which I was looking at
When this is my actual VM
b
Was there a live migration?
p
If I get the vmi object
Copy code
spec:
  affinity:
  domain:
    cpu:
      cores: 12
      dedicatedCpuPlacement: true
      model: Broadwell
      sockets: 1
      threads: 1
    devices:
      disks:
      interfaces:
    features:
      acpi:
        enabled: true
    firmware:
      uuid: afd43242-5daf-588f-ac6d-08a88fbf9868
    machine:
      type: q35
    memory:
      guest: 32256Mi
    resources:
      limits:
        cpu: "12"
        memory: 32Gi
      requests:
        cpu: "12"
        memory: 32Gi
Maybe there was a migration done a long time ago, but the VM has been restarted since.
b
Check the pods object
See if there two
p
No, just one. And its spec also has just 32GB. Now unless this is a problem on Prometheus/Grafana's side?
This is my overcommit config:
Copy code
{
  "cpu": 1600,
  "memory": 150,
  "storage": 200
}
Would the 
additional-guest-memory-overhead-ratio
have an impact on this?
b
Can you dump a describe pods on that virtual launcher
p
Copy code
Name:             virt-launcher-sicorax-hdtdd
Namespace:        sicorax
Priority:         0
Service Account:  default
Node:             harvester-2/10.0.1.62
Start Time:       Wed, 19 Nov 2025 07:17:18 +0000
Labels:           <http://harvesterhci.io/vmName=sicorax|harvesterhci.io/vmName=sicorax>
                  <http://kubevirt.io|kubevirt.io>=virt-launcher
                  <http://kubevirt.io/created-by=1997da23-fd9d-47e5-89c9-457ee503cfc9|kubevirt.io/created-by=1997da23-fd9d-47e5-89c9-457ee503cfc9>
                  <http://kubevirt.io/nodeName=harvester-2|kubevirt.io/nodeName=harvester-2>
                  <http://vm.kubevirt.io/name=sicorax|vm.kubevirt.io/name=sicorax>
Annotations:      <http://cni.projectcalico.org/containerID|cni.projectcalico.org/containerID>: e49657d171e3a6abc7d391cef739ea7c961388d25b4c75a7c3af34a3bc24d278
                  <http://cni.projectcalico.org/podIP|cni.projectcalico.org/podIP>: 10.52.1.229/32
                  <http://cni.projectcalico.org/podIPs|cni.projectcalico.org/podIPs>: 10.52.1.229/32
                  <http://harvesterhci.io/sshNames|harvesterhci.io/sshNames>: []
                  <http://k8s.v1.cni.cncf.io/network-status|k8s.v1.cni.cncf.io/network-status>:
                    [{
                        "name": "default/prod-ntwrk",
                        "interface": "pod37a8eec1ce1",
                        "mac": "be:06:86:8a:66:36",
                        "dns": {}
                    }]
                  <http://k8s.v1.cni.cncf.io/networks|k8s.v1.cni.cncf.io/networks>: [{"name":"prod-ntwrk","namespace":"default","mac":"be:06:86:8a:66:36","interface":"pod37a8eec1ce1"}]
                  <http://kubectl.kubernetes.io/default-container|kubectl.kubernetes.io/default-container>: compute
                  <http://kubevirt.io/domain|kubevirt.io/domain>: sicorax
                  <http://kubevirt.io/migrationTransportUnix|kubevirt.io/migrationTransportUnix>: true
                  <http://kubevirt.io/vm-generation|kubevirt.io/vm-generation>: 23
                  <http://post.hook.backup.velero.io/command|post.hook.backup.velero.io/command>: ["/usr/bin/virt-freezer", "--unfreeze", "--name", "sicorax", "--namespace", "sicorax"]
                  <http://post.hook.backup.velero.io/container|post.hook.backup.velero.io/container>: compute
                  <http://pre.hook.backup.velero.io/command|pre.hook.backup.velero.io/command>: ["/usr/bin/virt-freezer", "--freeze", "--name", "sicorax", "--namespace", "sicorax"]
                  <http://pre.hook.backup.velero.io/container|pre.hook.backup.velero.io/container>: compute
Status:           Running
IP:               10.52.1.229
IPs:
  IP:           10.52.1.229
Controlled By:  VirtualMachineInstance/sicorax
Init Containers:
  container-disk-binary:
    Container ID:  <containerd://3902edfb0663148a12cccb6325358513337952d7578bd364f42169f8d612683>1
    Image:         <http://registry.suse.com/suse/sles/15.5/virt-launcher:1.2.2-150500.8.21.1|registry.suse.com/suse/sles/15.5/virt-launcher:1.2.2-150500.8.21.1>
    Image ID:      sha256:2ccd4d88a5427a267fd7d3438a29aff10717f49e66b047e5f578c17b0bbabcb1
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/cp
      /usr/bin/container-disk
      /init/usr/bin/container-disk
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Nov 2025 07:17:22 +0000
      Finished:     Wed, 19 Nov 2025 07:17:23 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     10m
      memory:  40M
    Requests:
      cpu:     10m
      memory:  40M
    Environment:
      XDG_CACHE_HOME:   /var/run/kubevirt-private
      XDG_CONFIG_HOME:  /var/run/kubevirt-private
      XDG_RUNTIME_DIR:  /var/run
    Mounts:
      /init/usr/bin from virt-bin-share-dir (rw)
  volumevirtio-container-disk-init:
    Container ID:  <containerd://07ec1657454708c1bac03db937043bd6206f265065abcce23af8a842f3e45ed>0
    Image:         <http://registry.suse.com/suse/vmdp/vmdp:2.5.4.2|registry.suse.com/suse/vmdp/vmdp:2.5.4.2>
    Image ID:      sha256:90567499610d496f7973e5b483cefa18d99c2f3c8ef9771547c93ba476a7872e
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/container-disk
    Args:
      --no-op
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Nov 2025 07:17:25 +0000
      Finished:     Wed, 19 Nov 2025 07:17:25 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     10m
      memory:  40M
    Requests:
      cpu:                10m
      ephemeral-storage:  50M
      memory:             40M
    Environment:          <none>
    Mounts:
      /usr/bin from virt-bin-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks/container-disk-data/1997da23-fd9d-47e5-89c9-457ee503cfc9 from container-disks (rw)
Containers:
  compute:
    Container ID:  <containerd://b1920379538ffb256b7853172b5f132097b5e0c978fce4a39ea73f820a320ca>b
    Image:         <http://registry.suse.com/suse/sles/15.5/virt-launcher:1.2.2-150500.8.21.1|registry.suse.com/suse/sles/15.5/virt-launcher:1.2.2-150500.8.21.1>
    Image ID:      sha256:2ccd4d88a5427a267fd7d3438a29aff10717f49e66b047e5f578c17b0bbabcb1
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/virt-launcher-monitor
      --qemu-timeout
      247s
      --name
      sicorax
      --uid
      1997da23-fd9d-47e5-89c9-457ee503cfc9
      --namespace
      sicorax
      --kubevirt-share-dir
      /var/run/kubevirt
      --ephemeral-disk-dir
      /var/run/kubevirt-ephemeral-disks
      --container-disk-dir
      /var/run/kubevirt/container-disks
      --grace-period-seconds
      135
      --hook-sidecars
      0
      --ovmf-path
      /usr/share/OVMF
      --run-as-nonroot
    State:          Running
      Started:      Wed, 19 Nov 2025 07:17:26 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:                      12
      <http://devices.kubevirt.io/kvm|devices.kubevirt.io/kvm>:  1
      <http://devices.kubevirt.io/tun|devices.kubevirt.io/tun>:  1
      memory:                   36530290913
    Requests:
      cpu:                      12
      <http://devices.kubevirt.io/kvm|devices.kubevirt.io/kvm>:  1
      <http://devices.kubevirt.io/tun|devices.kubevirt.io/tun>:  1
      ephemeral-storage:        50M
      memory:                   36530290913
    Environment:
      XDG_CACHE_HOME:                  /var/run/kubevirt-private
      XDG_CONFIG_HOME:                 /var/run/kubevirt-private
      XDG_RUNTIME_DIR:                 /var/run
      KUBEVIRT_RESOURCE_NAME_default:  
      POD_NAME:                        virt-launcher-sicorax-hdtdd (v1:metadata.name)
    Mounts:
      /var/run/kubevirt from public (rw)
      /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw)
      /var/run/kubevirt-private from private (rw)
      /var/run/kubevirt/container-disks from container-disks (rw)
      /var/run/kubevirt/hotplug-disks from hotplug-disks (rw)
      /var/run/kubevirt/sockets from sockets (rw)
      /var/run/libvirt from libvirt-runtime (rw)
    Devices:
      /dev/app-disk from app-disk
      /dev/backup-disk from backup-disk
      /dev/db-disk from db-disk
      /dev/os-disk from os-disk
  volumevirtio-container-disk:
    Container ID:  <containerd://063f78eb9f36961c765d80d61ea799c047c85832d63595cab7dff8c626760ff>6
    Image:         <http://registry.suse.com/suse/vmdp/vmdp:2.5.4.2|registry.suse.com/suse/vmdp/vmdp:2.5.4.2>
    Image ID:      sha256:90567499610d496f7973e5b483cefa18d99c2f3c8ef9771547c93ba476a7872e
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/container-disk
    Args:
      --copy-path
      /var/run/kubevirt-ephemeral-disks/container-disk-data/1997da23-fd9d-47e5-89c9-457ee503cfc9/disk_4
    State:          Running
      Started:      Wed, 19 Nov 2025 07:17:27 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     10m
      memory:  40M
    Requests:
      cpu:                10m
      ephemeral-storage:  50M
      memory:             40M
    Environment:          <none>
    Mounts:
      /usr/bin from virt-bin-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks/container-disk-data/1997da23-fd9d-47e5-89c9-457ee503cfc9 from container-disks (rw)
  guest-console-log:
    Container ID:  <containerd://348edc5dbff0333c34d1ab01e06b00fa13b64e2460db0e6c452eb90946f0612>0
    Image:         <http://registry.suse.com/suse/sles/15.5/virt-launcher:1.2.2-150500.8.21.1|registry.suse.com/suse/sles/15.5/virt-launcher:1.2.2-150500.8.21.1>
    Image ID:      sha256:2ccd4d88a5427a267fd7d3438a29aff10717f49e66b047e5f578c17b0bbabcb1
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/virt-tail
    Args:
      --logfile
      /var/run/kubevirt-private/1997da23-fd9d-47e5-89c9-457ee503cfc9/virt-serial0-log
    State:          Running
      Started:      Wed, 19 Nov 2025 07:17:27 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     15m
      memory:  60M
    Requests:
      cpu:     15m
      memory:  60M
    Environment:
      VIRT_LAUNCHER_LOG_VERBOSITY:  2
    Mounts:
      /var/run/kubevirt-private from private (ro)
Readiness Gates:
  Type                                   Status
  <http://kubevirt.io/virtual-machine-unpaused|kubevirt.io/virtual-machine-unpaused>   True 
Conditions:
  Type                                   Status
  <http://kubevirt.io/virtual-machine-unpaused|kubevirt.io/virtual-machine-unpaused>   True 
  PodReadyToStartContainers              True 
  Initialized                            True 
  Ready                                  True 
  ContainersReady                        True 
  PodScheduled                           True 
Volumes:
  private:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  public:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  sockets:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  virt-bin-share-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  libvirt-runtime:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  ephemeral-disks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  container-disks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  os-disk:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sicorax-os-disk-5ui2g
    ReadOnly:   false
  db-disk:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sicorax-db-disk-0szau
    ReadOnly:   false
  app-disk:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sicorax-app-disk-0eczk
    ReadOnly:   false
  backup-disk:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sicorax-backup-disk-jdy0q
    ReadOnly:   false
  hotplug-disks:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   <unset>
QoS Class:       Guaranteed
Node-Selectors:  <http://cpu-model.node.kubevirt.io/Broadwell=true|cpu-model.node.kubevirt.io/Broadwell=true>
                 cpumanager=true
                 <http://kubernetes.io/arch=amd64|kubernetes.io/arch=amd64>
                 <http://kubevirt.io/schedulable=true|kubevirt.io/schedulable=true>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:          <none>
b
Or 
-oyaml
p
If I run 
k top pods
it shows the pod as using just 32GB
b
So maybe promethius is a liar
You can check with crictl on the host
p
Okay, I'll check that. I assume the Rancher UI also uses the Prometheus metrics in that case, to get the host metrics. For example, this one node has 2 VMs, each with 32GB RAM, but uses twice as much as one would expect...
And if I run top on the host, it seems to say the two VMs are using 10% each, which is 32GB. So yeah, I take it that Prometheus may be off by a bit
a
it looks the prometheus/grafana run a bit tricky
23 Views
Open in Slack
PreviousNext
Powered by