This message was deleted.

You could try the PXE method: PXE Boot Installation | Harvester

@acoustic-addition-45641 I've read about it, but it's far from decent state managment of phisical servers that metal3 offers. So I wandered if anyone maybe actually tried to use metal3 with harvester? 😆

Due to Harvester’s unique way of booting it has to be the HOST os. PXE is the way : You can automate the install fairly easy.

@thousands-advantage-10804 I understand the pxe method, just with ironic/metal3 you first automated the configuration of bare metal (bios etc) and at the end you acutally boot and image. Hence I'm wondering (and will be willing to experiment) could I possibly put here: https://github.com/metal3-io/baremetal-operator/blob/main/examples/demo-hosts.yaml#L164 the url to the harvester image and then just provide to the harvester inataller the configuration it requirers

That operator uses libvirt to actually boot “images” from a qcow2. Harvester is designed to be on the host itself. There us not a qcow2 for harvester available because the install process is vital. It won’t work.

@thousands-advantage-10804 in this example there is qcow2, but the docs also mention raw (I guess it's the same as ISO?) https://book.metal3.io/bmo/introduction.html#deploying-baremetalhosts. libvirt? I might be mistake, but as I tested the metal3 flow to provision baremetal hosts with ubuntu image I just configured the bios/redfish (some hardware RADIS etc) and then just told it to boot ubuntu, ending up with exacly the same situation as I would if I configured the bios by hand and then boot ubuntu form iso/usb

I would love to see if works. It looked like it may be able to manage the host through the BMC.

I would love to hear that somone else tried it 😄 if not I will try soon 😄

@fancy-fountain-55453 please share those findings. I've been playing with the Dell Terraform Redfish provider recently in an effort to baremetal boot our Dell servers to a fully provisioned host OS (Ubuntu primarily) without the need to involve PXE. The general idea is to have the provider mount the installation ISO and also mount a separate IMG that contains a cloud config (Harvester config) CIDATA volume. However, Harvester does not have an autoinstall option when booting the ISO, so automating a Harvester install in this manner isn't an option. What would be nice is if Harvester had an autoinstall countdown timer immediately after ISO boot ("Looking for autoinstall file. Press any key to proceed.") that would at least scan other mounts for the presence of cloud-config (Harvester config) CIDATA volume... or something like that. I'm pretty sure that Redfish APIs are not Dell specific, so the Terraform provider may be applicable to the BMC of other hardware platforms. The Dell Redfish Terraform provider also allows for adjusting Dell's server configuration profile, which allows for (as an example) booting a RedHat golden image and providing answer files. All that to say, I wish Harvester had a means of autoinstalling with just an ISO + cloud-config (Harvester config). @thousands-advantage-10804 Love your YouTube content! Keep it coming!!!

You need to make a video showing Redfish.

can't you just set

kargs

like

harvester.install.automatic=true harvester.install.config_url

etc and it will install from ISO? we've basically done this before in a pinch

@acoustic-addition-45641 when provisioning (any image) with matal3 you can inject a config file to for example configure cloud-init. (https://book.metal3.io/bmo/instance_customization#userdata) as i read the docs

User and network data are passed to the instance via a so called _config drive_, which is a small additional disk partition created on the root device during provisioning.

so it also boils down auto mounting some config as a separate partition.

how are you passing the config drive to harvester installer?

since this is living rent free in my head.. Is there a way to pass the userdata into https://book.metal3.io/bmo/live-iso

since this is living rent free in my head..

- I know the feeling 😄 today I mobilized people to fix the bare metal network in our small DC so I can experiment 😄 @thousands-advantage-10804 regarding extra kernel params I found this https://github.com/metal3-io/baremetal-operator/blob/4409f7185cef78b4162da8a4c5acb[…]546/config/base/crds/bases/metal3.io_preprovisioningimages.yaml I need to dig into this, but even if some option is lacking it's just pull request to metal3 away from beeing added 🙂

crazy thought. use metal3 to boot the install kernel with the args to pull the iso from http/tftp like pxe.

@acoustic-addition-45641 @thousands-advantage-10804 @prehistoric-morning-49258 so I did some experiments (I will share snipsets). First, this is a working provisioning of ubuntu (with some user and networkdata injected for cloud-init):

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: hdd-n1
spec:
  online: true
  bootMACAddress: '3c:ec:ef:aa:b4:78'
  bmc:
    address: <ipmi://172.20.1.21:623>
    credentialsName: hdd-n1-registration-secret
  image:
    checksumType: sha256
    checksum: <http://172.20.110.2:6180/images/SHA256SUMS>
    format: qcow2
    url: <http://172.20.110.2:6180/images/jammy-server-cloudimg-amd64.img>
  userData:
    name: user-data-secret
  networkData:
    name: hdd-n1-networkdata-secret

At the end of this, you get a bmh with all ssh keys injected, created users, manually assigned ip etc. As for harvester, after some trial and error, I arrived at:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: hdd-n1
spec:
  online: true
  bootMACAddress: '3c:ec:ef:aa:b4:78'
  bmc:
    address: <ipmi://172.20.1.21:623>
    credentialsName: hdd-n1-registration-secret
  image:
    # Harvester
    checksumType: sha512
    checksum: <http://172.20.110.2:6180/images/harvester-v1.6.0-rc6-amd64.sha512>
    format: live-iso
    url: <http://172.20.110.2:6180/images/harvester-v1.6.0-rc6-amd64.iso>

It does load the harvester iso, the grub screen (attached) and then it fails on some

/dracut-state.sh

failing to find

/dev/disk/by-label/COS_LIVE

disk. I guess it's just a matter of some debug to fix it. But There still remains a question that @acoustic-addition-45641 raised, how to supply the config to live iso install so that everything happens automatically. The possible (tho hacky way) is just to build on the PXE work of @thousands-advantage-10804, use the

netboot

as an image in bmh spec that will boot eveyrthing else like he shown in his video. The small benefit of integrating it with metal3 is that all PXE infra is already there as before the system iso is used metal3 uses PXE to load

ironic agent

to inspect the hardware, configure radis etc. So it's just the matter of to asking the agent to download a

netboot

is with

ironic agent

and the

netboot

PXE config can be stored in k8s config maps and mounted in the tftf folder as well, making it more automatic process and more gitops friendly. Anyway the metal3+netboot solution seems quite hacky. Possible solution lies here

<https://owlet.today/posts/integrating-coreos-installer-with-ironic/>

where

customDeploy

is used to customize the process, but I need to git into it more 🙂

IRONIC_KERNEL_PARAMS - Kernel parameters to pass to IPA (default console=ttyS0)

sounds like it ought to work

@prehistoric-morning-49258

IRONIC_KERNEL_PARAMS

but those parameters go to the IPA (ironic python agent) ramdisk that is first booted to inspect/configure baremetal, before actually OS happens. So far I failed to find a way to inject anything apart from mounted disk with

USER_DATA

and

NETWORK_DATA

which are later used by cloud-init and such.

something like ipxe should work. might need to bake an ISO per-server depending on reqs, and/or scripted takeover install. basically boot to whatever OS can chain/takeover into harvester installer, imo ipxe is usually best option

@acoustic-addition-45641 @prehistoric-morning-49258 @thousands-advantage-10804 so I tried to follow what you suggested and boot whatever OS and then harvester takes over, however the @thousands-advantage-10804 solution with

<http://netboot.xyz|netboot.xyz>

requirers DHCP server that points to netboot ipxe/tftf infra. That creates a conflict with ironic dhcp/ftft infra that is in place already. And I have no idea how double IPXE would be possible here (first ironic and ironic agent then netboot with harvester). Ironic/Metal3 in theory could at least load the harvester ISO (no automated install but at least let us have something running!) but according to metal3 doc (https://book.metal3.io/bmo/live-iso)

booted OS will not be able to access any data on the ISO except for the kernel and initramfs it booted from.

and I see errors of harvester not finding

/root

and

/dev/disk/by-label/COS_LIVE

paths. Atm. I'm out of ideas how to proceed. I wonder why there is no qcow2 for harvester?

We basically use something like this: https://github.com/tlhakhan/ipxe-iso to build custom ipxe ISO. that drops you into a shell:

RUN <<EOF cat > run.ipxe

#!ipxe

shell

EOF

or you can update to like:

#!ipxe

chain https://[my_server]/harvester-config.ipxe

should then be able to just boot ISO and have Harvester auto-install

oh damm, abort? start drinking?

@prehistoric-morning-49258 with just ipxe it's indeed easy, I just had a dream of combining with ironic/meta3 etc for more gitops managment 🙂 But ironic serves its own boot.ipxe (for provisioning of ipa agent to gather inventory of bare metal) that is picked by netboot and brakes the netboot process ;-(

idk its like metal3 is whats preventing it being automated

@prehistoric-morning-49258 well with metal3 you would have a nice gitops repo with all bare metal and it's configuration there just waiting to be boostraped (or reinstalled in case or early very iterative development process 🙂 ) with ipxe it's more interactive automation 🙂 as you have to do a bit of manual picking (which host to start harvester with, which to join etc). Also, I'm not sure how harvester handles server with a lot of disks. Normally, a pre-provisioning tool - like ipa - does mdadm magic, or metal3 talkis via redfish to configure the hardware raid before the os is booted. Here I wonder what are actual capabilities of harvester are for configuring the hardware... looking at this sole variable https://docs.harvesterhci.io/v1.1/install/harvester-configuration#installdata_disk not soo big 😉

yeah you can pre-bake in a variety of ways, really it seems more like a metal3 support question than harvester support