https://rancher.com/ logo
Title
l

loud-belgium-84462

08/15/2022, 5:06 PM
hi i was wondering what is the branching strategy for fleet. If I have x number of applications, should we have one repo for all the applications or should we be having x repo's corresponding to each applications. Any suggestions for fleet is welcome. Also, when we define gitrepo objects in rancher again should we put as one gitrepo or nultiple gitrepo objects
h

high-morning-12231

08/15/2022, 6:07 PM
This choice largely comes down to what business process you want to use and how you want developers to interact with the system.
I personally would use a git repo per component group with internal dependencies that need to be upgraded as a full set.
l

loud-belgium-84462

08/15/2022, 6:16 PM
thanks so much for reply. so for example i have a helm chat that deploys bunch of k8s manifest together that could be one potential component group. Is that correct statement. We could create git repo per helm chart that belongs to that particular component
by git repo i mean git repo object that belongs to fleet crd. regarding physical git repository, do we need to seggregate that as well or they can reside in one physical git repository
another example can be a frontend, backend and middleware yamls that all belong to an application. application = component group = helm chart
h

high-morning-12231

08/15/2022, 9:49 PM
It's hard to answer in a concrete way as it really depends on your constraints. I think my suggestion would be to start with the simplest setup you can make work and then break apart as you run into places where the process complexity pushes you to re-evaluate.
p

powerful-elephant-25838

08/16/2022, 8:08 AM
We use a twofold strategy: 1 for managing the cluster components (monitoring, logging etc) with only 1 gitrepo. 2 for projects with 1 gitrepo in local fleet that looks in a repo of gitrepo crd that contains project 's repo info so we dynamically add them with gitops and than the 1 gitrepo for the helm of the project instance
👍 1
l

loud-belgium-84462

08/16/2022, 4:55 PM
so lets say you have repo-a in bitbucket. You would create multiple folders in that repo-a let me call them monitoring,logging, etc etc. Then you define a gitrepo object in rancher fleet that points to repo-a and specify all the paths i.e folders in the repo. Is that correct assumption on strategy1
sorry i didnt get it can you please explain with simple example what it means
p

powerful-elephant-25838

08/16/2022, 9:25 PM
Yes that is the correct assumption: I have a gitlab repo called "infrastructure " and on different folders I have a bunch of fleet.yaml files (some with overlay folders since x a couple of things I need to apply kustomize or add some specific yaml) and in the gitrepo I have patches for each folder in the repo. This strategy to have dependencies since I believe works only within the context of a single gitrepo and for example I need to install sealed secrets before stuff that uses it
🤔 1
X the second case I have a gitrepo that point to a gitlab repo called "projects " in 'local' fleet where I place, using gitops, gitrepo yaml crd files. This allows me to have fleet local configure fleet. Each single project_x_repo.yaml is the gitrepo that points to the project_x fleet file. This way I can give the project control of its fleet file while keeping gitops on all env. Project x team will work only on the repo it owns. The repo where we keep fleet is also the one used for helm of the project x, just a separate folder. We also use branches for having fleet observe only the correct branch
One thing you need to consider is that everytime fleet feels a change it start messing with bundles and checking if there are changes... so ... for something like the cluster config it can be an option to use folders but for projects my advice is go for 1 gitrepo = 1 project
l

loud-belgium-84462

08/16/2022, 9:45 PM
you mean 1 gitrepo object that has a project repository in gitlab
when you say projects does that mean applications that run on k8s ?
p

powerful-elephant-25838

08/16/2022, 9:49 PM
We have teams that produce software apps x our customers: they get 1 repo for the helm of their app. In that repo we place a folder called 'fleet's where we place the fleet.yaml that the team have to maintain with the values.yaml file that overrides helm values (to manage different env or instances)
They do not manage the gitrepo needed to fleet but is managed by ops team that just add it on another repo with access only to ops
Since we like to automate as much as possible this last repo is managed by fleet that configure fleet : there is the first gitrepo in local fleet that point to this 'infrastructure ' repo and add the gitrepo.yaml to default fleet
l

loud-belgium-84462

08/16/2022, 9:54 PM
so in the repo they have helm that points to helm chart in tgz format and there is a fleet.yaml in the fleet folder. I am trying to understand what would trigger the gitops
p

powerful-elephant-25838

08/16/2022, 9:56 PM
Ok ... this is a bit messy but will try to explain also to get feedback from fleet people
I will have to also add some decision points that shape the solution:
1) we do not want to have too many repo around since devs would complain and make a mess
l

loud-belgium-84462

08/16/2022, 9:58 PM
true
p

powerful-elephant-25838

08/16/2022, 9:59 PM
2) we want to have a build pipeline x helm to check compliance etc
3) we need to use software we already have even if it could be an overkill for some aspects
l

loud-belgium-84462

08/16/2022, 10:01 PM
ok
a small diagram might help everyone. I am trying to create one based on our discussion
p

powerful-elephant-25838

08/16/2022, 10:09 PM
The helm repo for each project keep the yaml files for helm and the fleet files in 2 separate folders with their structure. Than at each commit we have gitlab awake jenkins that build the helm chart validate it etc and than place it in artifactory. Fleet will get the chart from there and not from the repo. The only problem we could have is that fleet will be quicker than Jenkins. But we never had this problem, on this my question to fleet team is if the bundle will retry to get the chart in case is not yet published. Should this problem arise we'll solve with a branch dedicated to fleet where we can separate the change of the chart release from the actual build
l

loud-belgium-84462

08/16/2022, 10:13 PM
so you are saying that the project repo does not link to the gitrepo at all. It doesnt wake up the gitops process.
in the project 1 repo for example we have helm chart and the fleet yaml for overriding the values for the helm chart. after the build is over the chart gets uploaded in artifactory
so you are suggesting somehow the second is connected to first
p

powerful-elephant-25838

08/16/2022, 10:16 PM
Let us make an example we have project 'dummy'
l

loud-belgium-84462

08/16/2022, 10:17 PM
okay
p

powerful-elephant-25838

08/16/2022, 10:18 PM
We have repo 'dummy_helm' with 2 folders 'fleet's and 'helm'
l

loud-belgium-84462

08/16/2022, 10:19 PM
yup
p

powerful-elephant-25838

08/16/2022, 10:20 PM
In the fleet folder there is fleet.yaml and values.yaml. in fleet we have the helm description that point to artifactory to the 'dummy' chart release 1.0 using the values.yaml file as overrides
l

loud-belgium-84462

08/16/2022, 10:22 PM
fleet.yaml*
p

powerful-elephant-25838

08/16/2022, 10:22 PM
In the 'infrastructure ' repo we have a 'dummy_gitrepo.yaml' that points to repo 'dummy_helm' path to fleet folder branch deploy
🤔 2
l

loud-belgium-84462

08/16/2022, 10:25 PM
helm:
chart: dummy.tgz
valuesFiles: values.yaml from dummy folder
this is the fleet.yaml in your first case
p

powerful-elephant-25838

08/16/2022, 10:27 PM
When the team tag a new release on production branch (1.1) gitlab awakes Jenkins that builds the chart and places it in artifactory
Yes apart from the valuesFiles that is not from dummy folder but fleet folder in dummy_repo
And chart should be just dummy without the since we get it from artifactory that is a chart catalogue so fleet will get it from there and you need the repo: part
Without the tgz
l

loud-belgium-84462

08/16/2022, 10:32 PM
helm: chart: arifactory/dummyhelmchart.tgz # A https URL to a Helm repo to download the chart from. It's typically easier # to just use
chart
field and refer to a tgz file. If repo is used the # value of
chart
will be used as the chart name to lookup in the Helm repository. releaseName: my-dummy # The version of the chart or semver constraint of the chart to find. If a constraint # is specified it is evaluated each time git changes. version: 1.0.0 # Any values that should be placed in the
values.yaml
and passed to helm during # install. valuesFiles: - dummyfolder/fleet/values1.yaml
helm: chart: dummy releaseName: my-dummy version: 1.0.0 valuesFiles: - dummyfolder/fleet/values1.yaml
p

powerful-elephant-25838

08/16/2022, 10:34 PM
We use repo since artifactory is a repository for helm charts so you need 'repo: http://'
Also no dummyfolder 😄
l

loud-belgium-84462

08/16/2022, 10:36 PM
helm: chart: dummy repo: http:// releaseName: my-dummy version: 1.0.0 valuesFiles: - fleet/values1.yaml
p

powerful-elephant-25838

08/16/2022, 10:36 PM
Yes
That is the fleet.yaml content in dummy_helm repo
l

loud-belgium-84462

08/16/2022, 10:37 PM
this is the fleet.yaml that sits in the fleet folder of dummy_helm repo 🙂
dummy_helm repo helm chart templates fleet fleet.yaml values.yaml
p

powerful-elephant-25838

08/16/2022, 10:40 PM
In this repo we use different branches 1 for fleet 1 for helm
l

loud-belgium-84462

08/16/2022, 10:41 PM
dummy_helm repo (contents) helm templates/ deployment.yaml service.yaml chart.yaml fleet fleet.yaml values.yaml
p

powerful-elephant-25838

08/16/2022, 10:42 PM
When the helm branch tag gitlab awakes the Jenkins pipeline and build the new chart
Correct the contents of repo
When we push on fleet branch we awake fleet
l

loud-belgium-84462

08/16/2022, 10:44 PM
let me call branch helm in dummy_helm repo and another branch called fleet in the repo
p

powerful-elephant-25838

08/16/2022, 10:44 PM
Ok
l

loud-belgium-84462

08/16/2022, 10:44 PM
contents of the branches are same
lets say we change deployment.yaml that results into a new helm chart in artifactory
p

powerful-elephant-25838

08/16/2022, 10:46 PM
Mostly .. they start the same but in time drift since developers will touch only the needed parts so helm branch will evolve with it and fleet will have only chart version changes and potentially values.yaml
l

loud-belgium-84462

08/16/2022, 10:47 PM
make sense
so developer has to adjust the fleet.yaml everytime there is a new version of chart
values.yaml says 3 replicas of the pod lets say
p

powerful-elephant-25838

08/16/2022, 10:48 PM
Flows is like this: on branch helm dev do changes to chart push and tag. Jenkins works fleet sleeps.
l

loud-belgium-84462

08/16/2022, 10:49 PM
poor jenkins 🙂
so now we have a new version of chart in artifactory
then developer goes in the fleet.yaml in the fleet folder adjust the version and push it
p

powerful-elephant-25838

08/16/2022, 10:50 PM
If all goes well and PM gives ok release gets promoted in production: devs work on fleet branch change chart version and values if needed and push and tag. Jenkins sleeps fleet work
l

loud-belgium-84462

08/16/2022, 10:53 PM
they change the chart version in the fleet.yaml to the latest chart that got deployed into artifactory
p

powerful-elephant-25838

08/16/2022, 10:53 PM
Yes
l

loud-belgium-84462

08/16/2022, 10:53 PM
helm: chart: dummychart16thaugustexample repo: http:// releaseName: my-dummy version: 1.0.0 valuesFiles: - fleet/values1.yaml
p

powerful-elephant-25838

08/16/2022, 10:54 PM
Or to the tag they want in production
l

loud-belgium-84462

08/16/2022, 10:54 PM
yup
how does fleet know that it has to wake up
p

powerful-elephant-25838

08/16/2022, 10:55 PM
In the gitrepo you can specify the branch also
l

loud-belgium-84462

08/16/2022, 10:55 PM
we have a gitrepo that is pointing to our dummy_helm repo
p

powerful-elephant-25838

08/16/2022, 10:55 PM
Yes we have
l

loud-belgium-84462

08/16/2022, 10:55 PM
ic so we have a gitrepo object that is pointing to fleet branch of the dummy _helm repo
and path is fleet
p

powerful-elephant-25838

08/16/2022, 10:56 PM
Yes and to fleet folder path
l

loud-belgium-84462

08/16/2022, 10:56 PM
yay 🙂
p

powerful-elephant-25838

08/16/2022, 10:56 PM
Correct
This way you have a single repo both for helm and fleet
l

loud-belgium-84462

08/16/2022, 10:58 PM
so next time there is a new project comes along lets say tesla we will have same structure meaning tesla_repo having helm and fleet subdirectory and a gitrepo that points to the fleet branch of the tesla_helm repo and path set to fleet
p

powerful-elephant-25838

08/16/2022, 10:58 PM
Since working with helm is not so messy we can use branches as a workaround for not having twice the repos
l

loud-belgium-84462

08/16/2022, 10:58 PM
ok
u said something about local fleet before what was that
p

powerful-elephant-25838

08/16/2022, 11:00 PM
Yes we forced convention also because we have automation that generate all this stuff
l

loud-belgium-84462

08/16/2022, 11:00 PM
ok
p

powerful-elephant-25838

08/16/2022, 11:01 PM
Is the way we manage all the gitrepos that points to the dummy_helm repos
l

loud-belgium-84462

08/16/2022, 11:01 PM
ic so you called that as local fleet
p

powerful-elephant-25838

08/16/2022, 11:01 PM
Option 1) you have an ops guy that write it ☹️
Option 2) you use fleet to deploy gitrepo crd into fleet itself
l

loud-belgium-84462

08/16/2022, 11:03 PM
confused about option2
are u saing we create the gitrepo crd object that does kubectl apply gitobject
saying*
p

powerful-elephant-25838

08/16/2022, 11:04 PM
You know that the gitrepo you add using rancher web interface is a crd
It has it's own yaml
l

loud-belgium-84462

08/16/2022, 11:05 PM
i didnt know that i just found out 2 sec ago 🙂
i was thinking a dummy like me would go in the gitrepos section of the ui and create that using gui
very itnteresting
so your automation will do that all that magic
p

powerful-elephant-25838

08/16/2022, 11:06 PM
So we keep all the yaml definitions for each project gitrepo in another gitlab repo called 'projects_gitrepos'
l

loud-belgium-84462

08/16/2022, 11:07 PM
ok
p

powerful-elephant-25838

08/16/2022, 11:07 PM
And we have fleet look at this repo so it can auto configure itself
So no more going to web ui
l

loud-belgium-84462

08/16/2022, 11:08 PM
ic
p

powerful-elephant-25838

08/16/2022, 11:09 PM
The only trick is that you have to know that there are 2 instances of fleet in rancher
🤔 1
l

loud-belgium-84462

08/16/2022, 11:09 PM
so in this projects_gitrepos we have gitrepos objects yaml for every application
p

powerful-elephant-25838

08/16/2022, 11:09 PM
Yes
l

loud-belgium-84462

08/16/2022, 11:09 PM
2 instances of fleet meaning?
p

powerful-elephant-25838

08/16/2022, 11:09 PM
All have to be in git
l

loud-belgium-84462

08/16/2022, 11:10 PM
i need few bevys and couple of them for you my frien
friend
can you please clarify what you mean by two instances of fleet
one that is reading developers fleet branches for all the projects
and fleet that is listening to the setup yamls in the projects_gitrepos doing setups
is the infrastructure repo different than the project_gitrepos that you mentioned
infrastructure repo(contents) monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/
so in addition to this structure the infra repo will have the gitrepo yamls that point to the project repos i.e dummy ones
infrastructure repo(contents) #infracomponents helmcharts##### monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/ ###gitrepocrdsobjectsofapplicationsproject#### gitrepoproject1.yaml gitrepoproject2.yaml
p

powerful-elephant-25838

08/17/2022, 7:03 AM
There is a fleet local that rancher uses to control the fleet instances in the clusters, but on this I ask help of fleet people here for better details
We solved the dev part, on the ops side we have 2 repos
First to manage cluster components called 'infrastructure' with folders like monitoring, logging, sealedsecrets, etc and, as we said this repo is watched by a gitrepo that uses many paths for each folder so we can use dependencies
Second the one with all the 'dummy_x_gitrepo.yaml' for the projects called 'projects_gitrepos' ; this is looked by a gitrepo from fleet local that will add the gitrepos to fleet as a way to use fleet to configure fleet
l

loud-belgium-84462

08/17/2022, 12:25 PM
Thanks for clarifying the ops repos in great detail
so you mentioned you have a repo called infra that has different folder underneath and then we have a gitrepo object that points to this rep and includes all the subfolders as the path(s). So within those subfolder lets say monitoring I am guessing you have helm directory that has the helm chart that deploys that component and a fleet directory that has fleet.yaml. you mentioned "I have a bunch of fleet.yaml files (some with overlay folders since x a couple of things I need to apply kustomize or add some specific yaml) and in the gitrepo I have patches for each folder in the repo. This strategy to have dependencies since I believe works only within the context of a single gitrepo and for example I need to install sealed secrets before stuff that uses it"
infrastructure repo(contents) #infracomponents helmcharts##### monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/
p

powerful-elephant-25838

08/17/2022, 4:00 PM
For the infrastructure I never use helm chart as files but always refer to the one on official repo
We use the values: spec of fleet to customize specific parameters
We cannot effort to maintain all chart for monitoring etc
Also most comes from rancher repo
l

loud-belgium-84462

08/17/2022, 4:08 PM
thanks. so in the subfolder for monitoring for example you have a fleet.yaml that points to the official helm chart repo of rancher. In the subfolder for external secrets you have fleet.yaml that points to helm chart for external secrets
p

powerful-elephant-25838

08/17/2022, 4:09 PM
To give you the list of the components we add in our clusters: from rancher repo monitoring, logging, longhorn; from respective official repo trident (netapp csi), traefik, velero, kubecost, metallb, sealedsecret, vault, external secrets.
👍 1
l

loud-belgium-84462

08/17/2022, 4:09 PM
also in these fleet.yamls you have different values for different environment
p

powerful-elephant-25838

08/17/2022, 4:09 PM
Exactly
l

loud-belgium-84462

08/17/2022, 4:10 PM
ok great
i can see that helm charts can be installed via fleet.yaml or via gitrepos pointing to paths that have helm chart or k8s manifests
p

powerful-elephant-25838

08/17/2022, 4:11 PM
Also since we use metallb we need to inform fleet that that is a dependency for traefik since we need to get the ip for the lb of the ingress controller
👍 1
l

loud-belgium-84462

08/17/2022, 4:11 PM
i think installing helm chart via fleet.yaml is most popular
p

powerful-elephant-25838

08/17/2022, 4:12 PM
Also we use kustomize in fleet to add some additional yaml to the official chart where needed
l

loud-belgium-84462

08/17/2022, 4:13 PM
so basically the subfolder(s) are acted upon by kustomize
how do you indicate dependencies in those folders lets say external secrets going before others and other stuff
p

powerful-elephant-25838

08/17/2022, 4:14 PM
For example where we need to add the traefik IngressRoute crd instead of the usual Ingress that you find in official charts
Fleet has 'dependency:' option
You have to state the name of the proper path that has to go before
👍 1
l

loud-belgium-84462

08/17/2022, 4:15 PM
ah ok that make sense
so basically you indicate via path what goes before the other
p

powerful-elephant-25838

08/17/2022, 4:17 PM
This way you can have a complete new cluster always coherent in about 15 minutes
👍 1
l

loud-belgium-84462

08/17/2022, 4:18 PM
its pretty cool
so this infrastructure has master branch that is tagged for uat and tag that deploys to production i suppose
just one infra repo deploying to all environment
p

powerful-elephant-25838

08/17/2022, 4:20 PM
My advice is not use the option that fleet hasto customize for different clusters group because this will generate a unnecessary fleet execution of fleet on all clusters for every push, better to use branches and different gitrepo for different clusters
So 1 repo and 1 branch for different clusters
This way for example we can customize metallb ip ranges for each cluster in the branch of the cluster
l

loud-belgium-84462

08/17/2022, 4:22 PM
if we have one uat cluster and 2 prod clusters we will have three repos namely infra_uat, infra_prod1, infra_prod2
and do not use clustergroup
p

powerful-elephant-25838

08/17/2022, 4:23 PM
No 1 repo and 3 branches
l

loud-belgium-84462

08/17/2022, 4:24 PM
ic in one repo you have dev, uat, prod branches and each branch is deploying to different environments/clusters
p

powerful-elephant-25838

08/17/2022, 4:25 PM
We also create and assigns clusters in cluster group 'just in case' ... I like to plan for what could happen but for now no use
Yes 1 branch 1 cluster
l

loud-belgium-84462

08/17/2022, 4:25 PM
if we have two prod clusters we will have two branches
fleet is generally running on management cluster that happens to be rancher cluster right, this rancher cluster can talk to uat, prod, or any other clusters i refer them to as downstream clusters
p

powerful-elephant-25838

08/17/2022, 4:28 PM
We discovered this and changed with a quirk on the trident CSI, at each push on different cluster groups we got all cluster have a redeploy of trident chart and this was not well received by pvc a bug somewhere so we abandoned custom deploy in fleet and went to branches
Yes that is the way fleet work, the fleet local runs on upstream rancher and commands fleet on downstream
Yes correct
l

loud-belgium-84462

08/17/2022, 5:13 PM
trying to put our conversation in a picture so others can benefit i hope rancher fleet people are also listening and they can create a blog post out of it 🙂
so the fleet local runs on upstream rancher and commands fleet on downstream are two different things as I understand?
i added projects repos as well
p

powerful-elephant-25838

08/17/2022, 5:34 PM
Will try to post a blog myself in these days
🙏 1
👍 1
l

loud-belgium-84462

08/17/2022, 6:58 PM
The approach you have mentioned here is generic and can be applied to any gitops tools such as argocd, flux and fleet
You mentioned about some patching mechanism for the subfolder in the infra repository. What was it for please
hi there just from a recap you mentioned we have one repo for infra that has multiple branches corresponding to each clusters. lets say we have ist environment, uat environment and prod environment each having 2 clusters so we will have in total three branches ist, uat and production branch
so lets say we merged a code change to ist that deploys to that environment then we test if everything is good we merge ist branch into uat that deploys to uat via gitops then we test everything is good then we merge into prod branch and delete uat. Does that look good to you