https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
rke2 includes calico (and the CRDs) already, is there any particular reason you’re not using that?
I’m not aware of anything that can’t be configured in the helm chart values, if there is something, open an issue on GH and we’ll see what we can do about it.
c
I'm a bit confused as well wit the mix between RKE & RKE2... your question is only related to RKE2 and not RKE, right ? And why are not using the probided Calico ?
v
Thank you for responding. To answer your questions in order:
(for some reason it wouldn’t let me edit my previous message) Thank you for responding. To answer your question, we started deploying calico via helm chart in RKE due to the most recent calico version in RKE being far from current. Even before that, we found we had to add custom resources for items like 
BGPConfig
(to add explicit service/externalIP CIDRs, control BGP communities, and support BGP passwords), 
BGPPeers
(to add multiple endpoints and node selectors), 
felixConfiguration
to control iptables/DSR behavior when testing eBPF, and custom 
IPPools
to control the resource naming to simplify troubleshooting. Now that we’re testing RKE2 I am curious if we can migrate to the instance that is provided, but we would need a mechanism for applying those custom resources, preferably as part of the same install so we can stop worrying about race conditions where the Rancher v3 API doesn’t include the machine record until after the CNI and network are functioning and cluster-registration has completed.
c
moving to RKE2 would be a wise move if you're interested to have recent versions of calico indeed. It gets regularly updated
👍 1
v
that is definitely part of our motivation, along with greater control over components like kube-proxy and whether they are installed at all
in this case, the challenge is mainly around order of operations and trying to simplify application of the complete config as much as possible
since these higher level calico CRDs are not included in the operator, I was just curious if there is a mechanism to provide them to RKE2 and have it 
kubectl apply -f
them so-to-speak during the install
totally understand if it isn’t possible, currently we just have to watch for kube-apiserver to come up on the first node, then run the commands directly for it to finish building the cluster and joining to Rancher UI
c
Can you describe what specifically your doing with your customizations? As I understand it most things should be done through the helm chart values. For anything else there is the option to deploy manifests that are applied when the server starts.
v
A good number of the config options aren’t present in the Tigera Operator chart, with the expectation that you simply apply them directly via kubectl/calicoctl either before or after the chart is deployed. In our case, the most important ones are 
BGPConfiguration
to define prefix advertisements and BGP communities, 
FelixConfiguration
to configure BPF behaviors, and 
BGPPeers
to add the explicit peers with node selectors. We also currently define an 
IPPool
directly via custom resource that matches the operator config options, but that is only to better control the naming (`cluster-node-ippool-10-10-1-0-24' vs 
default
).
c
Yeah, the Installation data at least can be put in the helm chart, but the BGPConfiguration cannot.
All you need to do to use the manifests directory is to drop the yaml files in that directory
v
just to be clear though, that will only handle the helm values, not the additional CRDs, correct?
c
https://docs.rke2.io/helm/#automatically-deploying-manifests-and-helm-charts
Any Kubernetes manifests found in 
/var/lib/rancher/rke2/server/manifests
will automatically be deployed to RKE2 in a manner similar to 
kubectl apply
. Manifests deployed in this manner are managed as AddOn custom resources, and can be viewed by running 
kubectl get addon -A
.
You can put a HelmChartConfig manifest in there and it will customize the packaged HelmChart. But as the docs say, you can put absolutely any valid manifest in there.
v
that was my initial question around the ramifications of being 
AddOn custom resources
and how those are processed
c
It’s just another custom resource type that can be used to track the status of applying each file in that directory. if you describe the addon you can see events when it is applied, errors if there are any, etc. If you have an RKE2 cluster available, go try it out. There’s not much to it.
v
sounds good, will give that a look
thanks for your help
13 Views
Open in Slack
PreviousNext
Powered by