https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# fleet
a
This message was deleted.
h
This choice largely comes down to what business process you want to use and how you want developers to interact with the system.
I personally would use a git repo per component group with internal dependencies that need to be upgraded as a full set.
l
thanks so much for reply. so for example i have a helm chat that deploys bunch of k8s manifest together that could be one potential component group. Is that correct statement. We could create git repo per helm chart that belongs to that particular component
by git repo i mean git repo object that belongs to fleet crd. regarding physical git repository, do we need to seggregate that as well or they can reside in one physical git repository
another example can be a frontend, backend and middleware yamls that all belong to an application. application = component group = helm chart
h
It's hard to answer in a concrete way as it really depends on your constraints. I think my suggestion would be to start with the simplest setup you can make work and then break apart as you run into places where the process complexity pushes you to re-evaluate.
p
We use a twofold strategy: 1 for managing the cluster components (monitoring, logging etc) with only 1 gitrepo. 2 for projects with 1 gitrepo in local fleet that looks in a repo of gitrepo crd that contains project 's repo info so we dynamically add them with gitops and than the 1 gitrepo for the helm of the project instance
👍 1
l
so lets say you have repo-a in bitbucket. You would create multiple folders in that repo-a let me call them monitoring,logging, etc etc. Then you define a gitrepo object in rancher fleet that points to repo-a and specify all the paths i.e folders in the repo. Is that correct assumption on strategy1
sorry i didnt get it can you please explain with simple example what it means
p
Yes that is the correct assumption: I have a gitlab repo called "infrastructure " and on different folders I have a bunch of fleet.yaml files (some with overlay folders since x a couple of things I need to apply kustomize or add some specific yaml) and in the gitrepo I have patches for each folder in the repo. This strategy to have dependencies since I believe works only within the context of a single gitrepo and for example I need to install sealed secrets before stuff that uses it
🤔 1
X the second case I have a gitrepo that point to a gitlab repo called "projects " in 'local' fleet where I place, using gitops, gitrepo yaml crd files. This allows me to have fleet local configure fleet. Each single project_x_repo.yaml is the gitrepo that points to the project_x fleet file. This way I can give the project control of its fleet file while keeping gitops on all env. Project x team will work only on the repo it owns. The repo where we keep fleet is also the one used for helm of the project x, just a separate folder. We also use branches for having fleet observe only the correct branch
One thing you need to consider is that everytime fleet feels a change it start messing with bundles and checking if there are changes... so ... for something like the cluster config it can be an option to use folders but for projects my advice is go for 1 gitrepo = 1 project
l
you mean 1 gitrepo object that has a project repository in gitlab
when you say projects does that mean applications that run on k8s ?
p
We have teams that produce software apps x our customers: they get 1 repo for the helm of their app. In that repo we place a folder called 'fleet's where we place the fleet.yaml that the team have to maintain with the values.yaml file that overrides helm values (to manage different env or instances)
They do not manage the gitrepo needed to fleet but is managed by ops team that just add it on another repo with access only to ops
Since we like to automate as much as possible this last repo is managed by fleet that configure fleet : there is the first gitrepo in local fleet that point to this 'infrastructure ' repo and add the gitrepo.yaml to default fleet
l
so in the repo they have helm that points to helm chart in tgz format and there is a fleet.yaml in the fleet folder. I am trying to understand what would trigger the gitops
p
Ok ... this is a bit messy but will try to explain also to get feedback from fleet people
I will have to also add some decision points that shape the solution:
1) we do not want to have too many repo around since devs would complain and make a mess
l
true
p
2) we want to have a build pipeline x helm to check compliance etc
3) we need to use software we already have even if it could be an overkill for some aspects
l
ok
a small diagram might help everyone. I am trying to create one based on our discussion
p
The helm repo for each project keep the yaml files for helm and the fleet files in 2 separate folders with their structure. Than at each commit we have gitlab awake jenkins that build the helm chart validate it etc and than place it in artifactory. Fleet will get the chart from there and not from the repo. The only problem we could have is that fleet will be quicker than Jenkins. But we never had this problem, on this my question to fleet team is if the bundle will retry to get the chart in case is not yet published. Should this problem arise we'll solve with a branch dedicated to fleet where we can separate the change of the chart release from the actual build
l
so you are saying that the project repo does not link to the gitrepo at all. It doesnt wake up the gitops process.
in the project 1 repo for example we have helm chart and the fleet yaml for overriding the values for the helm chart. after the build is over the chart gets uploaded in artifactory
so you are suggesting somehow the second is connected to first
p
Let us make an example we have project 'dummy'
l
okay
p
We have repo 'dummy_helm' with 2 folders 'fleet's and 'helm'
l
yup
p
In the fleet folder there is fleet.yaml and values.yaml. in fleet we have the helm description that point to artifactory to the 'dummy' chart release 1.0 using the values.yaml file as overrides
l
fleet.yaml*
p
In the 'infrastructure ' repo we have a 'dummy_gitrepo.yaml' that points to repo 'dummy_helm' path to fleet folder branch deploy
🤔 2
l
helm:
chart: dummy.tgz
valuesFiles: values.yaml from dummy folder
this is the fleet.yaml in your first case
p
When the team tag a new release on production branch (1.1) gitlab awakes Jenkins that builds the chart and places it in artifactory
Yes apart from the valuesFiles that is not from dummy folder but fleet folder in dummy_repo
And chart should be just dummy without the since we get it from artifactory that is a chart catalogue so fleet will get it from there and you need the repo: part
Without the tgz
l
helm: chart: arifactory/dummyhelmchart.tgz # A https URL to a Helm repo to download the chart from. It's typically easier # to just use 
chart
field and refer to a tgz file. If repo is used the # value of 
chart
will be used as the chart name to lookup in the Helm repository. releaseName: my-dummy # The version of the chart or semver constraint of the chart to find. If a constraint # is specified it is evaluated each time git changes. version: 1.0.0 # Any values that should be placed in the 
values.yaml
and passed to helm during # install. valuesFiles: - dummyfolder/fleet/values1.yaml
helm: chart: dummy releaseName: my-dummy version: 1.0.0 valuesFiles: - dummyfolder/fleet/values1.yaml
p
We use repo since artifactory is a repository for helm charts so you need 'repo: http://'
Also no dummyfolder 😄
l
helm: chart: dummy repo: http:// releaseName: my-dummy version: 1.0.0 valuesFiles: - fleet/values1.yaml
p
Yes
That is the fleet.yaml content in dummy_helm repo
l
this is the fleet.yaml that sits in the fleet folder of dummy_helm repo 🙂
dummy_helm repo helm chart templates fleet fleet.yaml values.yaml
p
In this repo we use different branches 1 for fleet 1 for helm
l
dummy_helm repo (contents) helm templates/ deployment.yaml service.yaml chart.yaml fleet fleet.yaml values.yaml
p
When the helm branch tag gitlab awakes the Jenkins pipeline and build the new chart
Correct the contents of repo
When we push on fleet branch we awake fleet
l
let me call branch helm in dummy_helm repo and another branch called fleet in the repo
p
Ok
l
contents of the branches are same
lets say we change deployment.yaml that results into a new helm chart in artifactory
p
Mostly .. they start the same but in time drift since developers will touch only the needed parts so helm branch will evolve with it and fleet will have only chart version changes and potentially values.yaml
l
make sense
so developer has to adjust the fleet.yaml everytime there is a new version of chart
values.yaml says 3 replicas of the pod lets say
p
Flows is like this: on branch helm dev do changes to chart push and tag. Jenkins works fleet sleeps.
l
poor jenkins 🙂
so now we have a new version of chart in artifactory
then developer goes in the fleet.yaml in the fleet folder adjust the version and push it
p
If all goes well and PM gives ok release gets promoted in production: devs work on fleet branch change chart version and values if needed and push and tag. Jenkins sleeps fleet work
l
they change the chart version in the fleet.yaml to the latest chart that got deployed into artifactory
p
Yes
l
helm: chart: dummychart16thaugustexample repo: http:// releaseName: my-dummy version: 1.0.0 valuesFiles: - fleet/values1.yaml
p
Or to the tag they want in production
l
yup
how does fleet know that it has to wake up
p
In the gitrepo you can specify the branch also
l
we have a gitrepo that is pointing to our dummy_helm repo
p
Yes we have
l
ic so we have a gitrepo object that is pointing to fleet branch of the dummy _helm repo
and path is fleet
p
Yes and to fleet folder path
l
yay 🙂
p
Correct
This way you have a single repo both for helm and fleet
l
so next time there is a new project comes along lets say tesla we will have same structure meaning tesla_repo having helm and fleet subdirectory and a gitrepo that points to the fleet branch of the tesla_helm repo and path set to fleet
p
Since working with helm is not so messy we can use branches as a workaround for not having twice the repos
l
ok
u said something about local fleet before what was that
p
Yes we forced convention also because we have automation that generate all this stuff
l
ok
p
Is the way we manage all the gitrepos that points to the dummy_helm repos
l
ic so you called that as local fleet
p
Option 1) you have an ops guy that write it ☹️
Option 2) you use fleet to deploy gitrepo crd into fleet itself
l
confused about option2
are u saing we create the gitrepo crd object that does kubectl apply gitobject
saying*
p
You know that the gitrepo you add using rancher web interface is a crd
It has it's own yaml
l
i didnt know that i just found out 2 sec ago 🙂
i was thinking a dummy like me would go in the gitrepos section of the ui and create that using gui
very itnteresting
so your automation will do that all that magic
p
So we keep all the yaml definitions for each project gitrepo in another gitlab repo called 'projects_gitrepos'
l
ok
p
And we have fleet look at this repo so it can auto configure itself
So no more going to web ui
l
ic
p
The only trick is that you have to know that there are 2 instances of fleet in rancher
🤔 1
l
so in this projects_gitrepos we have gitrepos objects yaml for every application
p
Yes
l
2 instances of fleet meaning?
p
All have to be in git
l
i need few bevys and couple of them for you my frien
friend
can you please clarify what you mean by two instances of fleet
one that is reading developers fleet branches for all the projects
and fleet that is listening to the setup yamls in the projects_gitrepos doing setups
is the infrastructure repo different than the project_gitrepos that you mentioned
infrastructure repo(contents) monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/
so in addition to this structure the infra repo will have the gitrepo yamls that point to the project repos i.e dummy ones
infrastructure repo(contents) #infracomponents helmcharts##### monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/ ###gitrepocrdsobjectsofapplicationsproject#### gitrepoproject1.yaml gitrepoproject2.yaml
p
There is a fleet local that rancher uses to control the fleet instances in the clusters, but on this I ask help of fleet people here for better details
We solved the dev part, on the ops side we have 2 repos
First to manage cluster components called 'infrastructure' with folders like monitoring, logging, sealedsecrets, etc and, as we said this repo is watched by a gitrepo that uses many paths for each folder so we can use dependencies
Second the one with all the 'dummy_x_gitrepo.yaml' for the projects called 'projects_gitrepos' ; this is looked by a gitrepo from fleet local that will add the gitrepos to fleet as a way to use fleet to configure fleet
l
Thanks for clarifying the ops repos in great detail
so you mentioned you have a repo called infra that has different folder underneath and then we have a gitrepo object that points to this rep and includes all the subfolders as the path(s). So within those subfolder lets say monitoring I am guessing you have helm directory that has the helm chart that deploys that component and a fleet directory that has fleet.yaml. you mentioned "I have a bunch of fleet.yaml files (some with overlay folders since x a couple of things I need to apply kustomize or add some specific yaml) and in the gitrepo I have patches for each folder in the repo. This strategy to have dependencies since I believe works only within the context of a single gitrepo and for example I need to install sealed secrets before stuff that uses it"
infrastructure repo(contents) #infracomponents helmcharts##### monitoring/ helm/ fleet/ fleet.yaml logging/ vault/ othercomponentsoftheplatform/
p
For the infrastructure I never use helm chart as files but always refer to the one on official repo
We use the values: spec of fleet to customize specific parameters
We cannot effort to maintain all chart for monitoring etc
Also most comes from rancher repo
l
thanks. so in the subfolder for monitoring for example you have a fleet.yaml that points to the official helm chart repo of rancher. In the subfolder for external secrets you have fleet.yaml that points to helm chart for external secrets
p
To give you the list of the components we add in our clusters: from rancher repo monitoring, logging, longhorn; from respective official repo trident (netapp csi), traefik, velero, kubecost, metallb, sealedsecret, vault, external secrets.
👍 1
l
also in these fleet.yamls you have different values for different environment
p
Exactly
l
ok great
i can see that helm charts can be installed via fleet.yaml or via gitrepos pointing to paths that have helm chart or k8s manifests
p
Also since we use metallb we need to inform fleet that that is a dependency for traefik since we need to get the ip for the lb of the ingress controller
👍 1
l
i think installing helm chart via fleet.yaml is most popular
p
Also we use kustomize in fleet to add some additional yaml to the official chart where needed
l
so basically the subfolder(s) are acted upon by kustomize
how do you indicate dependencies in those folders lets say external secrets going before others and other stuff
p
For example where we need to add the traefik IngressRoute crd instead of the usual Ingress that you find in official charts
Fleet has 'dependency:' option
You have to state the name of the proper path that has to go before
👍 1
l
ah ok that make sense
so basically you indicate via path what goes before the other
p
This way you can have a complete new cluster always coherent in about 15 minutes
👍 1
l
its pretty cool
so this infrastructure has master branch that is tagged for uat and tag that deploys to production i suppose
just one infra repo deploying to all environment
p
My advice is not use the option that fleet hasto customize for different clusters group because this will generate a unnecessary fleet execution of fleet on all clusters for every push, better to use branches and different gitrepo for different clusters
So 1 repo and 1 branch for different clusters
This way for example we can customize metallb ip ranges for each cluster in the branch of the cluster
l
if we have one uat cluster and 2 prod clusters we will have three repos namely infra_uat, infra_prod1, infra_prod2
and do not use clustergroup
p
No 1 repo and 3 branches
l
ic in one repo you have dev, uat, prod branches and each branch is deploying to different environments/clusters
p
We also create and assigns clusters in cluster group 'just in case' ... I like to plan for what could happen but for now no use
Yes 1 branch 1 cluster
l
if we have two prod clusters we will have two branches
fleet is generally running on management cluster that happens to be rancher cluster right, this rancher cluster can talk to uat, prod, or any other clusters i refer them to as downstream clusters
p
We discovered this and changed with a quirk on the trident CSI, at each push on different cluster groups we got all cluster have a redeploy of trident chart and this was not well received by pvc a bug somewhere so we abandoned custom deploy in fleet and went to branches
Yes that is the way fleet work, the fleet local runs on upstream rancher and commands fleet on downstream
l
i drew this quickly
p
Yes correct
l
trying to put our conversation in a picture so others can benefit i hope rancher fleet people are also listening and they can create a blog post out of it 🙂
so the fleet local runs on upstream rancher and commands fleet on downstream are two different things as I understand?
i added projects repos as well
p
Will try to post a blog myself in these days
🙏 1
👍 1
l
The approach you have mentioned here is generic and can be applied to any gitops tools such as argocd, flux and fleet
You mentioned about some patching mechanism for the subfolder in the infra repository. What was it for please
hi there just from a recap you mentioned we have one repo for infra that has multiple branches corresponding to each clusters. lets say we have ist environment, uat environment and prod environment each having 2 clusters so we will have in total three branches ist, uat and production branch
so lets say we merged a code change to ist that deploys to that environment then we test if everything is good we merge ist branch into uat that deploys to uat via gitops then we test everything is good then we merge into prod branch and delete uat. Does that look good to you
125 Views
Open in Slack
PreviousNext
Powered by