@rapid-van-91305 @astonishing-stone-14417 Trying to give some thoughts to the way we can get RKE2 binaries on the machines, that would also work with Air-Gapped environments: 1. Having a different image for each version of RKE2: is that at all imaginable? How are images downloaded in the final user's environment? 2. Using cURL in cloud-init's runcmd to get binaries from a local HTTP server (would be a pre-requisite from a user's side) 3. Supposing the user has put all necessary images in a private container registry (also a pre-requisite, and solves a part of the problem, some binaries like the RKE2 binary still need to be transferred some how). I was already expecting this but indeed, secrets and ConfigMaps are limited to 1MiB size, and therefore are not designed for hosting files on the API. Since CAPI uses cloud-init, we don't have anyway to check in advance if node is air-gapped (e.g. checking if github.com resolves and check if we get a valid HTTP response within a short time frame) before installing RKE2, except maybe if we do a lot of bash scripting and run it using cloud-init, which should not be the goal of CAPI I imagine. Do you have thoughts about this ?

With the Kubeadm bootstrap provider the binaries are baked into the base image

That also means there is an image version for each binary version

We could also support downloading, given a url in the spec and as you say download as part of cloud-init

I like the "baked into image" approach though it puts more pressure on project maintenance, because it does NOT suppose pre-requisites from the user's perspective

Its how CAPI currently works and is accepted

Well, the project maintenance overhead can be limited using proper automation

Agreed. We can adopt image-builder potentially….or whatever we currently use at SUSE (obs?) to build images

Yes

Having the option of downloading from an artefact server / registry is also a nice option

How does the standard image land on the user's environment ?

Depends on the infra provider being used. Some infra providers have a way to look up images / standard download location. Also most providers have a way to specify a custom image

Oh... so we just reference an image ID or something, and the infra provider figures out how to get our image ...

I’d say, that we publish images in a variety of formats with RKE2 builtin and then supply instructions on how to obtain/use those images.

I would have liked to support that, as probably most of the community settings will NOT be air-gapped, but I see only two ways of doing both air-gapped and non-air-gapped: • We put in the API and explicit option for Air-Gapped environments which would create a completely different cloud-init config. • We do some magic in a complex bash script that is run by cloud-init to figured out if the node is air-gapped or not, and continue with the right approach...

If we support both baked into image and download from a specified location we should be able to cover most scenarios.

yes, I only fear the various scenarios users would want us to implement for their remote location download: • SSL with a custom CA • Basic Auth • Other Auth • S3 buckets • ...

Good point.

We can have 2 things supported first: • baked into image • completely non-air-gapped : download from GitHub public URLs

That could work. Although i’m wondering if we just do 1 to start with and then think about download later

Sure, that's what Orange did for their fork

Baked in?

yes

Good, that is the capi way 🙂

OK then! Let's do that!

So we start with issue 24 🙂 Shall we create a discussion in the repo around downloading

Makes sense to me

Do you have an example / reference I can look into ?

For CAPI images: https://github.com/kubernetes-sigs/image-builder/tree/master/images/capi

no that's it

or maybe customizing it so we can reuse upstream code without forks

In my last company we used it as a submodule and built on top of it

adding stuff on top

yes that was generally my suggestion, since submission upstream might take some time it could be some temp hack until you do it 'right' along the submission of the provider upstream

I don’t think it would be temp to be honest 🙂

but I mean ready for an upstream submission only when submitting the rest of the work

👍