https://rancher.com/ logo
Title
c

cool-thailand-26552

11/15/2022, 11:45 AM
@rapid-van-91305 @astonishing-stone-14417 Trying to give some thoughts to the way we can get RKE2 binaries on the machines, that would also work with Air-Gapped environments: 1. Having a different image for each version of RKE2: is that at all imaginable? How are images downloaded in the final user's environment? 2. Using cURL in cloud-init's runcmd to get binaries from a local HTTP server (would be a pre-requisite from a user's side) 3. Supposing the user has put all necessary images in a private container registry (also a pre-requisite, and solves a part of the problem, some binaries like the RKE2 binary still need to be transferred some how). I was already expecting this but indeed, secrets and ConfigMaps are limited to 1MiB size, and therefore are not designed for hosting files on the API. Since CAPI uses cloud-init, we don't have anyway to check in advance if node is air-gapped (e.g. checking if github.com resolves and check if we get a valid HTTP response within a short time frame) before installing RKE2, except maybe if we do a lot of bash scripting and run it using cloud-init, which should not be the goal of CAPI I imagine. Do you have thoughts about this ?
r

rapid-van-91305

11/15/2022, 11:46 AM
With the Kubeadm bootstrap provider the binaries are baked into the base image
This is a good option
c

cool-thailand-26552

11/15/2022, 11:47 AM
That also means there is an image version for each binary version
👍 1
r

rapid-van-91305

11/15/2022, 11:47 AM
We could also support downloading, given a url in the spec and as you say download as part of cloud-init
Yep, if you bake the dependencies into a image then there will be an image for each rke2 version
c

cool-thailand-26552

11/15/2022, 11:49 AM
I like the "baked into image" approach though it puts more pressure on project maintenance, because it does NOT suppose pre-requisites from the user's perspective
r

rapid-van-91305

11/15/2022, 11:49 AM
Its how CAPI currently works and is accepted
But if people want custom images then we need to supply a list of things they need to include
c

cool-thailand-26552

11/15/2022, 11:50 AM
Well, the project maintenance overhead can be limited using proper automation
r

rapid-van-91305

11/15/2022, 11:50 AM
Agreed. We can adopt image-builder potentially….or whatever we currently use at SUSE (obs?) to build images
c

cool-thailand-26552

11/15/2022, 11:50 AM
Yes
r

rapid-van-91305

11/15/2022, 11:51 AM
Having the option of downloading from an artefact server / registry is also a nice option
c

cool-thailand-26552

11/15/2022, 11:51 AM
How does the standard image land on the user's environment ?
r

rapid-van-91305

11/15/2022, 11:52 AM
Depends on the infra provider being used. Some infra providers have a way to look up images / standard download location. Also most providers have a way to specify a custom image
c

cool-thailand-26552

11/15/2022, 11:53 AM
Oh... so we just reference an image ID or something, and the infra provider figures out how to get our image ...
r

rapid-van-91305

11/15/2022, 11:54 AM
I’d say, that we publish images in a variety of formats with RKE2 builtin and then supply instructions on how to obtain/use those images.
But we can also support the download option as well as that will be fully contained in our provider 🙂
c

cool-thailand-26552

11/15/2022, 11:57 AM
I would have liked to support that, as probably most of the community settings will NOT be air-gapped, but I see only two ways of doing both air-gapped and non-air-gapped: • We put in the API and explicit option for Air-Gapped environments which would create a completely different cloud-init config. • We do some magic in a complex bash script that is run by cloud-init to figured out if the node is air-gapped or not, and continue with the right approach...
r

rapid-van-91305

11/15/2022, 11:59 AM
If we support both baked into image and download from a specified location we should be able to cover most scenarios.
Air gapped or not
With air gapped guessing we need the container images as well or ability to set different registry / proxy
c

cool-thailand-26552

11/15/2022, 12:00 PM
yes, I only fear the various scenarios users would want us to implement for their remote location download: • SSL with a custom CA • Basic Auth • Other Auth • S3 buckets • ...
r

rapid-van-91305

11/15/2022, 12:00 PM
Good point.
Lets start with the baked into image approach (like capi) then?
We can then think about downloading later?
c

cool-thailand-26552

11/15/2022, 12:01 PM
We can have 2 things supported first: • baked into image • completely non-air-gapped : download from GitHub public URLs
what do you think ?
r

rapid-van-91305

11/15/2022, 12:03 PM
That could work. Although i’m wondering if we just do 1 to start with and then think about download later
Keep it simple initially to get to poc
c

cool-thailand-26552

11/15/2022, 12:03 PM
Sure, that's what Orange did for their fork
r

rapid-van-91305

11/15/2022, 12:03 PM
Baked in?
c

cool-thailand-26552

11/15/2022, 12:03 PM
yes
r

rapid-van-91305

11/15/2022, 12:04 PM
Good, that is the capi way 🙂
c

cool-thailand-26552

11/15/2022, 12:05 PM
OK then! Let's do that!
r

rapid-van-91305

11/15/2022, 12:05 PM
So we start with issue 24 🙂 Shall we create a discussion in the repo around downloading
c

cool-thailand-26552

11/15/2022, 12:05 PM
Makes sense to me
c

careful-piano-35019

11/15/2022, 2:09 PM
Do you have an example / reference I can look into ?
r

rapid-van-91305

11/15/2022, 2:13 PM
Or did you mean their actual usage?
c

careful-piano-35019

11/15/2022, 2:13 PM
no that's it
so what about forking that repo to add capbr ?
a

astonishing-stone-14417

11/15/2022, 2:15 PM
or maybe customizing it so we can reuse upstream code without forks
we might have conflicts later because of carry patches
r

rapid-van-91305

11/15/2022, 2:17 PM
In my last company we used it as a submodule and built on top of it
👍 1
@astonishing-stone-14417 - when you say customize, do you mean customizing the project directly? or using it and adding customization on top?
a

astonishing-stone-14417

11/15/2022, 2:26 PM
adding stuff on top
c

careful-piano-35019

11/15/2022, 2:27 PM
yes that was generally my suggestion, since submission upstream might take some time it could be some temp hack until you do it 'right' along the submission of the provider upstream
r

rapid-van-91305

11/15/2022, 2:30 PM
I don’t think it would be temp to be honest 🙂
c

careful-piano-35019

11/15/2022, 2:31 PM
but I mean ready for an upstream submission only when submitting the rest of the work
(you could only build images for AWS first, and take care of different kind of images later)
r

rapid-van-91305

11/15/2022, 2:32 PM
👍
Depending on the project it can be hard to get what is perceived as vendor specific stuff accepted. But we can cross that bridge later 🙂