I feel completely let down with release v1.6.0 it is still using SLE Micro 5 / kernel 5.x (SLE Micro 6 postponed again to v1.7.0 https://github.com/harvester/harvester/issues/7025). So there is not functional Thunderbold (for external NVME and graphics), there is not UBLK driver ...

In fairness it looks like they made that choice back in April. The switch from wicked to NetworkManger is probably going to break a lot of workflows.

Yes, in Apr25 to 1.7.0 ☹️ but it was 1.6.0 in Nov24. So I expected this release and yes there is need to redesign NetworkManager + selinux (apparmor is not supported) ... But it is blocking features from LongHorn (UBLK driver is in kernel 6) and kernel 6 new drivers. Either way I do not understood SUSE decisions to use non-LTS kernel (5.14 instead 5.15) and (6.4 instead 6.1/6.6/6.12) 😢

Yeah I think they could also bump to 5.17, but I am not a kernel dev and I have no idea what would go into that.

(only 5.14 and 6.4) https://www.suse.com/support/kb/doc/?id=000021672

Well I think once they get over the major hurdel of moving to 6.x subsequent increases (6.2, etc.) will be more trivial.

I also don't understand why Rancher/SUSE ignores ZFS. It would be great to be able to use snapshot/rollback/clone/promote for rootfs instead of the strange "/oem," as well as use ZFS as the basis for rke2/containerd instead of overlayfs. The last time I used k3s from the Rancher/SUSE line, I found that ZFS support had been explicitly disabled, so I used Debian with ZFS and normal containerd as a base (https://github.com/k3s-io/k3s/issues/66).

I'm using harvester+zfs, for data not rootfs. it does all basically work. pretty sweet imo

Every use of ZFS demonstrates how filesystems should be done in the 21st century, rather than using outdated file systems from the last century. 😄 🤣

The hairy part of the SL Micro 6.x upgrade so far is the wicked -> networkmanager migration, and also the way that this affects upgrades. I'm working on this at the moment. I too wish it had been possible to get the SL Micro 6.x migration done sooner than it's happening, but we will get there. And, yes, subsequent Micro 6.x bumps should be more straightforward than this one.

Thanks for the pointer @ambitious-daybreak-95996. Yes, ZFS (=OpenZFS) is CDDL, CDDL (file licence) is compatible with GPL, but GPL (project licence) is only compatible with GPL 🫤. However I think many distributions are starting to ignore this minor issue (I am not a lawyer) and are using ZFS (=OpenZFS) (check PROXMOX). In my opinion is that ZFS is generally used for mission-critical servers and enterprises. (I set up my first ZFS in 2005). I tested "openSUSE Leap Micro 6.2 (raw)" and found the following: • kernel 6.12.0 LTS! 👍 • actual thunderbolt (TB) drivers 👍 • pciutils and usbutils must be installed manually. • integration of TB is bad, missing packages bolt + bolt-tools (can be installed from (leap/16.0/repo/oss/) or auto-authorise can be resolved with udev

ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"

, kernel+udev is NOT activated by hotplug (I NEVER seen this behavior), so I have to restart the scan manually to attach TB

echo "1" > /sys/bus/pci/rescan

😵‍💫 •

kernel/drivers/block/ublk_drv.ko.zst

is in

kernel-default-extra

and must be installed manually

I haven't tried this (but have made a note to do so), but you should be able to specify arbitrary bond options if you provide them to the installer in a config file, e.g.

install:
  ...
  bond_options:
    mode: active-backup
    miimon: 100
    xmit_hash_policy: layer3+4

...but this has to be done during initial installation.

/oem/harvester.config

is written out by the installer, and contains the configuration at install time. Editing it later won't actually change anything, and you're right, you have to make subsequent changes via YAML.

Yes, I found it out and also it generates more problems like to extend installer and so on... So thise leads me to question does "/oem" will be used in Leap/Suse micro 6.x (btrfs snapshots system with "transactional-update" instead of "overlayfs") ? "/etc" is normal (and snapshoted) rw fs, for example:

ID 256 gen 11 top level 5 path @
ID 257 gen 305 top level 256 path @/.snapshots
ID 258 gen 53 top level 257 path @/.snapshots/1/snapshot
ID 259 gen 22 top level 256 path @/home
ID 260 gen 22 top level 256 path @/opt
ID 261 gen 587 top level 256 path @/root
ID 262 gen 22 top level 256 path @/srv
ID 263 gen 589 top level 256 path @/var
ID 264 gen 22 top level 256 path @/boot/writable
ID 265 gen 587 top level 256 path @/usr/local
ID 266 gen 22 top level 256 path @/boot/grub2/i386-pc
ID 267 gen 22 top level 256 path @/boot/grub2/x86_64-efi
ID 268 gen 72 top level 258 path @/.snapshots/1/snapshot/etc
ID 269 gen 585 top level 263 path @/var/lib/machines
ID 270 gen 165 top level 257 path @/.snapshots/2/snapshot
ID 271 gen 182 top level 270 path @/.snapshots/2/snapshot/etc
ID 272 gen 300 top level 257 path @/.snapshots/3/snapshot
ID 273 gen 582 top level 272 path @/.snapshots/3/snapshot/etc
ID 275 gen 211 top level 257 path @/.snapshots/4/snapshot
ID 276 gen 588 top level 275 path etc

With harvester on sl micro 6, we'll continue to use the /oem directory, and are not switching to btrfs/transactional-update - we'll remain with elemental's active/passive image as we have now.

So extended questions for trasition to 6.x 🙂 I cannot find any relevant issue discussion in https://github.com/harvester/harvester/issues : • "/etc" stays "ro+overlayfs" and partially patched with "cloud-init/stages/initramfs" ?

overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/overlay/etc.overlay/upper,workdir=/run/overlay/etc.overlay/work)
/dev/nvme1n1p5 on /etc/systemd type ext4 (rw,relatime)
/dev/nvme1n1p5 on /etc/rancher type ext4 (rw,relatime)
/dev/nvme1n1p5 on /etc/ssh type ext4 (rw,relatime)
/dev/nvme1n1p5 on /etc/iscsi type ext4 (rw,relatime)
/dev/nvme1n1p5 on /etc/nvme type ext4 (rw,relatime)
/dev/nvme1n1p5 on /etc/cni type ext4 (rw,relatime)
/dev/nvme1n1p5 on /etc/pki/trust/anchors type ext4 (rw,relatime)

• other fs mountpoint stays in actual design (/usr/local/.state/*.bind -> mount) so no btrfs ? • containerd snapshotter stays "overlayfs" or will be switched to "btrfs" (https://github.com/containerd/containerd/issues/6067, https://github.com/containerd/containerd/issues/4217 is resolved ???) ? • ...

all of that stuff will remain as-is

Ok, thanks.

Hello we did cover @loud-potato-42814's feedback on SLE Micro extended core meeting today. internal slides https://docs.google.com/presentation/d/1U6ixkVCjY3_t4lHs2G98FesoXoh0um3h0fe2YtzhwKs/edit?slide=id.g37de30b8020_0_28#slide=id.g37de30b8020_0_28 Feature tracker for Leap MIcro code.opensuse.org/leap/features/issue/242 SLE Micro https://jira.suse.com/browse/SMO-590 Harvester implementation of SMO-590 https://github.com/harvester/harvester/issues/9118 And the performance issue with NVMe on Arm https://bugzilla.suse.com/show_bug.cgi?id=1249456

Thanks, @calm-ghost-68769 thanks. Maybe it can leads to move from kernel 5.14 (micro 5.5) to modern LTS kernel 6.12 (micro 6.2). Today I begin to test Longhorn 1.10.0-rc2 (still unsuccessful) maybe because it is recommended kernel 6.8 🙁 (not available in Harvester).