We've been having a lot of trouble with VMs that a...
# harvesterb
bland-article-62755
10/14/2025, 3:45 PMWe've been having a lot of trouble with VMs that are using ceph rbd volumes, and I'm wondering if anyone else is using the ceph rbd plugin to mount disks into vms even as data drives. We had been using them pretty early on (1.2.x) without any real issues, but recently (1.4.3) the VMs started crashing with IO errors.
bland-article-62755
10/14/2025, 3:48 PMWe did a ton a of ceph read-writes earlier this year with scraping/rescuing gov datasets with not that much trouble, but after recent updates it seems like the ceph kernel module is crashing on the host side.
bland-article-62755
10/14/2025, 3:49 PMThe mounts will go sideways and can't be removed, then later the virt-launcher pod zombies and can't be killed.
bland-article-62755
10/14/2025, 3:51 PMOften the node will fail to reboot at all and just throw tons of libceph errors:
bland-article-62755
10/14/2025, 3:51 PMimage.png
bland-article-62755
10/14/2025, 3:53 PMI'd think that maybe a newer kernel might help, but there's no kernel change from 1.4.3 -> 1.5.x or if that's even the cause.
bland-article-62755
10/31/2025, 3:28 PMI saw this KB Article and now I'm wondering if it's related because the RBD/ceph mounts are held or managed at the hypervisor/host/node level. We have like .75 TiBi of Ram on these boxes so I think they qualify for large RAM.
👀 1
bland-article-62755
10/31/2025, 4:01 PM Copy code
kube10:~ # cat /proc/sys/vm/dirty_ratio
20bland-article-62755
10/31/2025, 4:01 PMDef set to 20% ratio on 1.4.3
10 Views