https://rancher.com/ logo
Powered by
Title
v

victorious-analyst-3332

05/12/2022, 12:59 AM
Would anyone be able to tell me if it is possible to install arbitrary CRDs as a part of the RKE2 install process? We’re currently installing calico via Helm chart to stay more recent on RKE installs, but it appears that we can do most of that via RKE2 Helm Integration. In our case, we need to apply the calico CRDs and create a number of custom resources for things like BGP peering and felixConfig before the operator deploy (options not present in the operator), so I was hoping there might be a method for also applying the custom config. I did see the 
/var/lib/rancher/rke2/server/manifests
approach, but don’t fully understand the ramifications of the 
AddOn custom resources
detail mentioned there. Thanks a lot for any help you can give.
c

creamy-pencil-82913

05/12/2022, 6:52 AM
rke2 includes calico (and the CRDs) already, is there any particular reason you’re not using that?
I’m not aware of anything that can’t be configured in the helm chart values, if there is something, open an issue on GH and we’ll see what we can do about it.
c

careful-piano-35019

05/12/2022, 7:13 AM
I'm a bit confused as well wit the mix between RKE & RKE2... your question is only related to RKE2 and not RKE, right ? And why are not using the probided Calico ?
v

victorious-analyst-3332

05/12/2022, 12:02 PM
Thank you for responding. To answer your questions in order:
(for some reason it wouldn’t let me edit my previous message) Thank you for responding. To answer your question, we started deploying calico via helm chart in RKE due to the most recent calico version in RKE being far from current. Even before that, we found we had to add custom resources for items like 
BGPConfig
(to add explicit service/externalIP CIDRs, control BGP communities, and support BGP passwords), 
BGPPeers
(to add multiple endpoints and node selectors), 
felixConfiguration
to control iptables/DSR behavior when testing eBPF, and custom 
IPPools
to control the resource naming to simplify troubleshooting. Now that we’re testing RKE2 I am curious if we can migrate to the instance that is provided, but we would need a mechanism for applying those custom resources, preferably as part of the same install so we can stop worrying about race conditions where the Rancher v3 API doesn’t include the machine record until after the CNI and network are functioning and cluster-registration has completed.
c

careful-piano-35019

05/12/2022, 12:12 PM
moving to RKE2 would be a wise move if you're interested to have recent versions of calico indeed. It gets regularly updated
👍 1
v

victorious-analyst-3332

05/12/2022, 12:13 PM
that is definitely part of our motivation, along with greater control over components like kube-proxy and whether they are installed at all
in this case, the challenge is mainly around order of operations and trying to simplify application of the complete config as much as possible
since these higher level calico CRDs are not included in the operator, I was just curious if there is a mechanism to provide them to RKE2 and have it 
kubectl apply -f
them so-to-speak during the install
totally understand if it isn’t possible, currently we just have to watch for kube-apiserver to come up on the first node, then run the commands directly for it to finish building the cluster and joining to Rancher UI
c

creamy-pencil-82913

05/12/2022, 2:30 PM
Can you describe what specifically your doing with your customizations? As I understand it most things should be done through the helm chart values. For anything else there is the option to deploy manifests that are applied when the server starts.
v

victorious-analyst-3332

05/12/2022, 2:47 PM
A good number of the config options aren’t present in the Tigera Operator chart, with the expectation that you simply apply them directly via kubectl/calicoctl either before or after the chart is deployed. In our case, the most important ones are 
BGPConfiguration
to define prefix advertisements and BGP communities, 
FelixConfiguration
to configure BPF behaviors, and 
BGPPeers
to add the explicit peers with node selectors. We also currently define an 
IPPool
directly via custom resource that matches the operator config options, but that is only to better control the naming (`cluster-node-ippool-10-10-1-0-24' vs 
default
).
calico_crds_yml.yaml
tigera_operator_yml.yaml
c

creamy-pencil-82913

05/12/2022, 7:34 PM
Yeah, the Installation data at least can be put in the helm chart, but the BGPConfiguration cannot.
All you need to do to use the manifests directory is to drop the yaml files in that directory
v

victorious-analyst-3332

05/12/2022, 7:35 PM
just to be clear though, that will only handle the helm values, not the additional CRDs, correct?
c

creamy-pencil-82913

05/12/2022, 7:36 PM
https://docs.rke2.io/helm/#automatically-deploying-manifests-and-helm-charts
Any Kubernetes manifests found in 
/var/lib/rancher/rke2/server/manifests
will automatically be deployed to RKE2 in a manner similar to 
kubectl apply
. Manifests deployed in this manner are managed as AddOn custom resources, and can be viewed by running 
kubectl get addon -A
.
You can put a HelmChartConfig manifest in there and it will customize the packaged HelmChart. But as the docs say, you can put absolutely any valid manifest in there.
v

victorious-analyst-3332

05/12/2022, 7:37 PM
that was my initial question around the ramifications of being 
AddOn custom resources
and how those are processed
c

creamy-pencil-82913

05/12/2022, 7:38 PM
It’s just another custom resource type that can be used to track the status of applying each file in that directory. if you describe the addon you can see events when it is applied, errors if there are any, etc. If you have an RKE2 cluster available, go try it out. There’s not much to it.
v

victorious-analyst-3332

05/12/2022, 7:39 PM
sounds good, will give that a look
thanks for your help
7 Views
#rke2Join Slack
Powered by