hi i was wondering what is the branching strategy for fleet rancher-users #fleet

Title

# fleet

loud-belgium-84462

08/15/2022, 5:06 PM

hi i was wondering what is the branching strategy for fleet. If I have x number of applications, should we have one repo for all the applications or should we be having x repo's corresponding to each applications. Any suggestions for fleet is welcome. Also, when we define gitrepo objects in rancher again should we put as one gitrepo or nultiple gitrepo objects

high-morning-12231

08/15/2022, 6:07 PM

This choice largely comes down to what business process you want to use and how you want developers to interact with the system.

I personally would use a git repo per component group with internal dependencies that need to be upgraded as a full set.

loud-belgium-84462

08/15/2022, 6:16 PM

thanks so much for reply. so for example i have a helm chat that deploys bunch of k8s manifest together that could be one potential component group. Is that correct statement. We could create git repo per helm chart that belongs to that particular component

by git repo i mean git repo object that belongs to fleet crd. regarding physical git repository, do we need to seggregate that as well or they can reside in one physical git repository

another example can be a frontend, backend and middleware yamls that all belong to an application. application = component group = helm chart

high-morning-12231

08/15/2022, 9:49 PM

It's hard to answer in a concrete way as it really depends on your constraints. I think my suggestion would be to start with the simplest setup you can make work and then break apart as you run into places where the process complexity pushes you to re-evaluate.

powerful-elephant-25838

08/16/2022, 8:08 AM

We use a twofold strategy: 1 for managing the cluster components (monitoring, logging etc) with only 1 gitrepo. 2 for projects with 1 gitrepo in local fleet that looks in a repo of gitrepo crd that contains project 's repo info so we dynamically add them with gitops and than the 1 gitrepo for the helm of the project instance

👍 1

loud-belgium-84462

08/16/2022, 4:55 PM

so lets say you have repo-a in bitbucket. You would create multiple folders in that repo-a let me call them monitoring,logging, etc etc. Then you define a gitrepo object in rancher fleet that points to repo-a and specify all the paths i.e folders in the repo. Is that correct assumption on strategy1

sorry i didnt get it can you please explain with simple example what it means

powerful-elephant-25838

08/16/2022, 9:25 PM

Yes that is the correct assumption: I have a gitlab repo called "infrastructure " and on different folders I have a bunch of fleet.yaml files (some with overlay folders since x a couple of things I need to apply kustomize or add some specific yaml) and in the gitrepo I have patches for each folder in the repo. This strategy to have dependencies since I believe works only within the context of a single gitrepo and for example I need to install sealed secrets before stuff that uses it

🤔 1

X the second case I have a gitrepo that point to a gitlab repo called "projects " in 'local' fleet where I place, using gitops, gitrepo yaml crd files. This allows me to have fleet local configure fleet. Each single project_x_repo.yaml is the gitrepo that points to the project_x fleet file. This way I can give the project control of its fleet file while keeping gitops on all env. Project x team will work only on the repo it owns. The repo where we keep fleet is also the one used for helm of the project x, just a separate folder. We also use branches for having fleet observe only the correct branch

One thing you need to consider is that everytime fleet feels a change it start messing with bundles and checking if there are changes... so ... for something like the cluster config it can be an option to use folders but for projects my advice is go for 1 gitrepo = 1 project

loud-belgium-84462

08/16/2022, 9:45 PM

you mean 1 gitrepo object that has a project repository in gitlab

when you say projects does that mean applications that run on k8s ?

powerful-elephant-25838

08/16/2022, 9:49 PM

We have teams that produce software apps x our customers: they get 1 repo for the helm of their app. In that repo we place a folder called 'fleet's where we place the fleet.yaml that the team have to maintain with the values.yaml file that overrides helm values (to manage different env or instances)

They do not manage the gitrepo needed to fleet but is managed by ops team that just add it on another repo with access only to ops

Since we like to automate as much as possible this last repo is managed by fleet that configure fleet : there is the first gitrepo in local fleet that point to this 'infrastructure ' repo and add the gitrepo.yaml to default fleet

loud-belgium-84462

08/16/2022, 9:54 PM

so in the repo they have helm that points to helm chart in tgz format and there is a fleet.yaml in the fleet folder. I am trying to understand what would trigger the gitops

powerful-elephant-25838

08/16/2022, 9:56 PM

Ok ... this is a bit messy but will try to explain also to get feedback from fleet people

I will have to also add some decision points that shape the solution:

1) we do not want to have too many repo around since devs would complain and make a mess

loud-belgium-84462

08/16/2022, 9:58 PM

true

powerful-elephant-25838

08/16/2022, 9:59 PM

2) we want to have a build pipeline x helm to check compliance etc

3) we need to use software we already have even if it could be an overkill for some aspects

loud-belgium-84462

08/16/2022, 10:01 PM

a small diagram might help everyone. I am trying to create one based on our discussion

powerful-elephant-25838

08/16/2022, 10:09 PM

The helm repo for each project keep the yaml files for helm and the fleet files in 2 separate folders with their structure. Than at each commit we have gitlab awake jenkins that build the helm chart validate it etc and than place it in artifactory. Fleet will get the chart from there and not from the repo. The only problem we could have is that fleet will be quicker than Jenkins. But we never had this problem, on this my question to fleet team is if the bundle will retry to get the chart in case is not yet published. Should this problem arise we'll solve with a branch dedicated to fleet where we can separate the change of the chart release from the actual build

loud-belgium-84462

08/16/2022, 10:13 PM

so you are saying that the project repo does not link to the gitrepo at all. It doesnt wake up the gitops process.

in the project 1 repo for example we have helm chart and the fleet yaml for overriding the values for the helm chart. after the build is over the chart gets uploaded in artifactory

so you are suggesting somehow the second is connected to first

powerful-elephant-25838

08/16/2022, 10:16 PM

Let us make an example we have project 'dummy'

loud-belgium-84462

08/16/2022, 10:17 PM

okay

powerful-elephant-25838

08/16/2022, 10:18 PM

We have repo 'dummy_helm' with 2 folders 'fleet's and 'helm'

loud-belgium-84462

08/16/2022, 10:19 PM

yup

powerful-elephant-25838

08/16/2022, 10:20 PM

In the fleet folder there is fleet.yaml and values.yaml. in fleet we have the helm description that point to artifactory to the 'dummy' chart release 1.0 using the values.yaml file as overrides

loud-belgium-84462

08/16/2022, 10:22 PM

fleet.yaml*

powerful-elephant-25838

08/16/2022, 10:22 PM

In the 'infrastructure ' repo we have a 'dummy_gitrepo.yaml' that points to repo 'dummy_helm' path to fleet folder branch deploy

🤔 2

loud-belgium-84462

08/16/2022, 10:25 PM

helm:

chart: dummy.tgz

valuesFiles: values.yaml from dummy folder

this is the fleet.yaml in your first case

powerful-elephant-25838

08/16/2022, 10:27 PM

When the team tag a new release on production branch (1.1) gitlab awakes Jenkins that builds the chart and places it in artifactory

Yes apart from the valuesFiles that is not from dummy folder but fleet folder in dummy_repo

And chart should be just dummy without the since we get it from artifactory that is a chart catalogue so fleet will get it from there and you need the repo: part

Without the tgz

loud-belgium-84462

08/16/2022, 10:32 PM

helm: chart: arifactory/dummyhelmchart.tgz # A https URL to a Helm repo to download the chart from. It's typically easier # to just use

chart

field and refer to a tgz file. If repo is used the # value of

chart

will be used as the chart name to lookup in the Helm repository. releaseName: my-dummy # The version of the chart or semver constraint of the chart to find. If a constraint # is specified it is evaluated each time git changes. version: 1.0.0 # Any values that should be placed in the

values.yaml

and passed to helm during # install. valuesFiles: - dummyfolder/fleet/values1.yaml

helm: chart: dummy releaseName: my-dummy version: 1.0.0 valuesFiles: - dummyfolder/fleet/values1.yaml

powerful-elephant-25838

08/16/2022, 10:34 PM

We use repo since artifactory is a repository for helm charts so you need 'repo: http://'

Also no dummyfolder 😄

loud-belgium-84462

08/16/2022, 10:36 PM

helm: chart: dummy repo: http:// releaseName: my-dummy version: 1.0.0 valuesFiles: - fleet/values1.yaml

powerful-elephant-25838

08/16/2022, 10:36 PM

Yes

That is the fleet.yaml content in dummy_helm repo

loud-belgium-84462

08/16/2022, 10:37 PM

this is the fleet.yaml that sits in the fleet folder of dummy_helm repo 🙂

dummy_helm repo helm chart templates fleet fleet.yaml values.yaml

powerful-elephant-25838

08/16/2022, 10:40 PM

In this repo we use different branches 1 for fleet 1 for helm

loud-belgium-84462

08/16/2022, 10:41 PM

dummy_helm repo (contents) helm templates/ deployment.yaml service.yaml chart.yaml fleet fleet.yaml values.yaml

powerful-elephant-25838

08/16/2022, 10:42 PM

When the helm branch tag gitlab awakes the Jenkins pipeline and build the new chart

Correct the contents of repo

When we push on fleet branch we awake fleet

loud-belgium-84462

08/16/2022, 10:44 PM

let me call branch helm in dummy_helm repo and another branch called fleet in the repo

powerful-elephant-25838

08/16/2022, 10:44 PM

loud-belgium-84462

08/16/2022, 10:44 PM

contents of the branches are same

lets say we change deployment.yaml that results into a new helm chart in artifactory

powerful-elephant-25838

08/16/2022, 10:46 PM

Mostly .. they start the same but in time drift since developers will touch only the needed parts so helm branch will evolve with it and fleet will have only chart version changes and potentially values.yaml

loud-belgium-84462

08/16/2022, 10:47 PM

make sense

so developer has to adjust the fleet.yaml everytime there is a new version of chart

values.yaml says 3 replicas of the pod lets say

powerful-elephant-25838

08/16/2022, 10:48 PM

Flows is like this: on branch helm dev do changes to chart push and tag. Jenkins works fleet sleeps.

loud-belgium-84462

08/16/2022, 10:49 PM

poor jenkins 🙂

so now we have a new version of chart in artifactory

then developer goes in the fleet.yaml in the fleet folder adjust the version and push it

powerful-elephant-25838

08/16/2022, 10:50 PM

If all goes well and PM gives ok release gets promoted in production: devs work on fleet branch change chart version and values if needed and push and tag. Jenkins sleeps fleet work

loud-belgium-84462

08/16/2022, 10:53 PM

they change the chart version in the fleet.yaml to the latest chart that got deployed into artifactory

powerful-elephant-25838

08/16/2022, 10:53 PM

Yes

loud-belgium-84462

08/16/2022, 10:53 PM

helm: chart: dummychart16thaugustexample repo: http:// releaseName: my-dummy version: 1.0.0 valuesFiles: - fleet/values1.yaml

powerful-elephant-25838

08/16/2022, 10:54 PM

Or to the tag they want in production

loud-belgium-84462

08/16/2022, 10:54 PM

yup

how does fleet know that it has to wake up

powerful-elephant-25838

08/16/2022, 10:55 PM

In the gitrepo you can specify the branch also

loud-belgium-84462

08/16/2022, 10:55 PM

we have a gitrepo that is pointing to our dummy_helm repo

powerful-elephant-25838

08/16/2022, 10:55 PM

Yes we have

loud-belgium-84462

08/16/2022, 10:55 PM

ic so we have a gitrepo object that is pointing to fleet branch of the dummy _helm repo

and path is fleet

powerful-elephant-25838

08/16/2022, 10:56 PM

Yes and to fleet folder path

loud-belgium-84462

08/16/2022, 10:56 PM

yay 🙂

powerful-elephant-25838

08/16/2022, 10:56 PM

Correct

This way you have a single repo both for helm and fleet

loud-belgium-84462

08/16/2022, 10:58 PM

so next time there is a new project comes along lets say tesla we will have same structure meaning tesla_repo having helm and fleet subdirectory and a gitrepo that points to the fleet branch of the tesla_helm repo and path set to fleet

powerful-elephant-25838

08/16/2022, 10:58 PM

Since working with helm is not so messy we can use branches as a workaround for not having twice the repos

loud-belgium-84462

08/16/2022, 10:58 PM

u said something about local fleet before what was that

powerful-elephant-25838

08/16/2022, 11:00 PM

Yes we forced convention also because we have automation that generate all this stuff

loud-belgium-84462

08/16/2022, 11:00 PM

powerful-elephant-25838

08/16/2022, 11:01 PM

Is the way we manage all the gitrepos that points to the dummy_helm repos

loud-belgium-84462

08/16/2022, 11:01 PM

ic so you called that as local fleet

powerful-elephant-25838

08/16/2022, 11:01 PM

Option 1) you have an ops guy that write it ☹️

Option 2) you use fleet to deploy gitrepo crd into fleet itself

loud-belgium-84462

08/16/2022, 11:03 PM

confused about option2

are u saing we create the gitrepo crd object that does kubectl apply gitobject

saying*

powerful-elephant-25838

08/16/2022, 11:04 PM

You know that the gitrepo you add using rancher web interface is a crd

It has it's own yaml

loud-belgium-84462

08/16/2022, 11:05 PM

i didnt know that i just found out 2 sec ago 🙂

i was thinking a dummy like me would go in the gitrepos section of the ui and create that using gui

very itnteresting

so your automation will do that all that magic

powerful-elephant-25838

08/16/2022, 11:06 PM

So we keep all the yaml definitions for each project gitrepo in another gitlab repo called 'projects_gitrepos'

loud-belgium-84462

08/16/2022, 11:07 PM

powerful-elephant-25838

08/16/2022, 11:07 PM

And we have fleet look at this repo so it can auto configure itself

So no more going to web ui

loud-belgium-84462

08/16/2022, 11:08 PM

powerful-elephant-25838

08/16/2022, 11:09 PM

The only trick is that you have to know that there are 2 instances of fleet in rancher

🤔 1

loud-belgium-84462

08/16/2022, 11:09 PM

so in this projects_gitrepos we have gitrepos objects yaml for every application

powerful-elephant-25838

08/16/2022, 11:09 PM

Yes

loud-belgium-84462

08/16/2022, 11:09 PM

2 instances of fleet meaning?

powerful-elephant-25838

08/16/2022, 11:09 PM

All have to be in git

loud-belgium-84462

08/16/2022, 11:10 PM

i need few bevys and couple of them for you my frien

friend

can you please clarify what you mean by two instances of fleet

one that is reading developers fleet branches for all the projects

and fleet that is listening to the setup yamls in the projects_gitrepos doing setups

is the infrastructure repo different than the project_gitrepos that you mentioned

infrastructure repo(contents) monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/

so in addition to this structure the infra repo will have the gitrepo yamls that point to the project repos i.e dummy ones

infrastructure repo(contents) #infracomponents helmcharts##### monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/ ###gitrepocrdsobjectsofapplicationsproject#### gitrepoproject1.yaml gitrepoproject2.yaml

powerful-elephant-25838

08/17/2022, 7:03 AM

There is a fleet local that rancher uses to control the fleet instances in the clusters, but on this I ask help of fleet people here for better details

We solved the dev part, on the ops side we have 2 repos

First to manage cluster components called 'infrastructure' with folders like monitoring, logging, sealedsecrets, etc and, as we said this repo is watched by a gitrepo that uses many paths for each folder so we can use dependencies

Second the one with all the 'dummy_x_gitrepo.yaml' for the projects called 'projects_gitrepos' ; this is looked by a gitrepo from fleet local that will add the gitrepos to fleet as a way to use fleet to configure fleet

loud-belgium-84462

08/17/2022, 12:25 PM

Thanks for clarifying the ops repos in great detail

so you mentioned you have a repo called infra that has different folder underneath and then we have a gitrepo object that points to this rep and includes all the subfolders as the path(s). So within those subfolder lets say monitoring I am guessing you have helm directory that has the helm chart that deploys that component and a fleet directory that has fleet.yaml. you mentioned "I have a bunch of fleet.yaml files (some with overlay folders since x a couple of things I need to apply kustomize or add some specific yaml) and in the gitrepo I have patches for each folder in the repo. This strategy to have dependencies since I believe works only within the context of a single gitrepo and for example I need to install sealed secrets before stuff that uses it"

infrastructure repo(contents) #infracomponents helmcharts##### monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/

powerful-elephant-25838

08/17/2022, 4:00 PM

For the infrastructure I never use helm chart as files but always refer to the one on official repo

We use the values: spec of fleet to customize specific parameters

We cannot effort to maintain all chart for monitoring etc

Also most comes from rancher repo

loud-belgium-84462

08/17/2022, 4:08 PM

thanks. so in the subfolder for monitoring for example you have a fleet.yaml that points to the official helm chart repo of rancher. In the subfolder for external secrets you have fleet.yaml that points to helm chart for external secrets

powerful-elephant-25838

08/17/2022, 4:09 PM

To give you the list of the components we add in our clusters: from rancher repo monitoring, logging, longhorn; from respective official repo trident (netapp csi), traefik, velero, kubecost, metallb, sealedsecret, vault, external secrets.

👍 1

loud-belgium-84462

08/17/2022, 4:09 PM

also in these fleet.yamls you have different values for different environment

powerful-elephant-25838

08/17/2022, 4:09 PM

Exactly

loud-belgium-84462

08/17/2022, 4:10 PM

ok great

i can see that helm charts can be installed via fleet.yaml or via gitrepos pointing to paths that have helm chart or k8s manifests

powerful-elephant-25838

08/17/2022, 4:11 PM

Also since we use metallb we need to inform fleet that that is a dependency for traefik since we need to get the ip for the lb of the ingress controller

👍 1

loud-belgium-84462

08/17/2022, 4:11 PM

i think installing helm chart via fleet.yaml is most popular

powerful-elephant-25838

08/17/2022, 4:12 PM

Also we use kustomize in fleet to add some additional yaml to the official chart where needed

loud-belgium-84462

08/17/2022, 4:13 PM

so basically the subfolder(s) are acted upon by kustomize

how do you indicate dependencies in those folders lets say external secrets going before others and other stuff

powerful-elephant-25838

08/17/2022, 4:14 PM

For example where we need to add the traefik IngressRoute crd instead of the usual Ingress that you find in official charts

Fleet has 'dependency:' option

You have to state the name of the proper path that has to go before

👍 1

loud-belgium-84462

08/17/2022, 4:15 PM

ah ok that make sense

so basically you indicate via path what goes before the other

powerful-elephant-25838

08/17/2022, 4:17 PM

This way you can have a complete new cluster always coherent in about 15 minutes

👍 1

loud-belgium-84462

08/17/2022, 4:18 PM

its pretty cool

so this infrastructure has master branch that is tagged for uat and tag that deploys to production i suppose

just one infra repo deploying to all environment

powerful-elephant-25838

08/17/2022, 4:20 PM

My advice is not use the option that fleet hasto customize for different clusters group because this will generate a unnecessary fleet execution of fleet on all clusters for every push, better to use branches and different gitrepo for different clusters

So 1 repo and 1 branch for different clusters

This way for example we can customize metallb ip ranges for each cluster in the branch of the cluster

loud-belgium-84462

08/17/2022, 4:22 PM

if we have one uat cluster and 2 prod clusters we will have three repos namely infra_uat, infra_prod1, infra_prod2

and do not use clustergroup

powerful-elephant-25838

08/17/2022, 4:23 PM

No 1 repo and 3 branches

loud-belgium-84462

08/17/2022, 4:24 PM

ic in one repo you have dev, uat, prod branches and each branch is deploying to different environments/clusters

powerful-elephant-25838

08/17/2022, 4:25 PM

We also create and assigns clusters in cluster group 'just in case' ... I like to plan for what could happen but for now no use

Yes 1 branch 1 cluster

loud-belgium-84462

08/17/2022, 4:25 PM

if we have two prod clusters we will have two branches

fleet is generally running on management cluster that happens to be rancher cluster right, this rancher cluster can talk to uat, prod, or any other clusters i refer them to as downstream clusters

powerful-elephant-25838

08/17/2022, 4:28 PM

We discovered this and changed with a quirk on the trident CSI, at each push on different cluster groups we got all cluster have a redeploy of trident chart and this was not well received by pvc a bug somewhere so we abandoned custom deploy in fleet and went to branches

Yes that is the way fleet work, the fleet local runs on upstream rancher and commands fleet on downstream

Yes correct

loud-belgium-84462

08/17/2022, 5:13 PM

trying to put our conversation in a picture so others can benefit i hope rancher fleet people are also listening and they can create a blog post out of it 🙂

so the fleet local runs on upstream rancher and commands fleet on downstream are two different things as I understand?

i added projects repos as well

powerful-elephant-25838

08/17/2022, 5:34 PM

Will try to post a blog myself in these days

🙏 1

👍 1

loud-belgium-84462

08/17/2022, 6:58 PM

The approach you have mentioned here is generic and can be applied to any gitops tools such as argocd, flux and fleet

You mentioned about some patching mechanism for the subfolder in the infra repository. What was it for please

hi there just from a recap you mentioned we have one repo for infra that has multiple branches corresponding to each clusters. lets say we have ist environment, uat environment and prod environment each having 2 clusters so we will have in total three branches ist, uat and production branch

so lets say we merged a code change to ist that deploys to that environment then we test if everything is good we merge ist branch into uat that deploys to uat via gitops then we test everything is good then we merge into prod branch and delete uat. Does that look good to you

64 Views