Hi, We are using Rancher + Fleet (GitOps) + k3s in a relatively large setup (multi-cluster + 100s of nodes). Something that we struggle with is feedback, e.g lack of information flow back to Github (where our GitOps repo is based). Today I was reading the documentation (😲), and stumbled over this:

The

fleet-agent

will continue to monitor the application bundle and report statuses back in the following order: bundledeployment > bundle > GitRepo > cluster.

(https://fleet.rancher.io/examples/) Is the fleet-agent able to report back to GitRepo? Is there any additional information on this? Appricate your support and feedback.

I see Fleet supports using the OCI format for Helm charts. Can Fleet reference a private Container Registry (Azure Container Registry) for the OCI charts? And will Fleet support signed OCI container with Notary v1 and/or v2?

Hi everyone, ... I need to configure fleet imageScan when using image name into helm chart values.yaml instead of raw kubernetes manifests. Someone could give me some example ? Thanks ;)

I am finding a behavior from Fleet very buggy, I deployed I test it deploying this fleet example https://github.com/rancher/fleet-examples/tree/master/single-cluster/helm It deployes well, I was even able to add a label from a value on fleet.yaml or values.yaml Then I tried using variableName: global.fleet.clusterLabels.LABELNAME on fleet.yaml as per documentation and on the helm chart {{ .Values.LABELNAME }} it did not work, even worst, after than the bundle or cluster doesnt update, no matter what I do

Hello. I am looking to deploy multiple helm charts as part of a Fleet deployment. If I understand correctly, a

fleet.yaml

file can only reference a single chart. So that if i want multiple helm releases (for different charts) I have to have multiple directories, each with their own

fleet.yaml

file? Is that accurate? For context, these are not inline charts.

Hello everyone, I am getting the following error on my fleet-agents after upgrading from 2.5.16 to 2.6.8.

Failed to register agent: looking up secret cattle-fleet-local-system/fleet-agent-bootstrap: Unauthorized

I have done some research and found when I run this

kubectl auth can-i get secret --as=system:serviceaccount:cattle-fleet-system:fleet-agent -n cattle-fleet-system

I get "no" and instead of "yes" Running this does show I have the secrets in place kubectl get secret -n cattle-fleet-system NAME TYPE DATA AGE c-9rmhj kubernetes.io/tls 2 43h default-token-x24mk kubernetes.io/service-account-token 3 43h fleet-controller-bootstrap-token-2lfsx kubernetes.io/service-account-token 3 43h fleet-controller-token-9gdts kubernetes.io/service-account-token 3 43h gitjob-token-r5jmr kubernetes.io/service-account-token 3 43h sh.helm.release.v1.fleet-crd.v1 helm.sh/release.v1 1 43h sh.helm.release.v1.fleet-crd.v2 helm.sh/release.v1 1 40h sh.helm.release.v1.fleet.v1 helm.sh/release.v1 1 43h sh.helm.release.v1.fleet.v2 helm.sh/release.v1 1 43h sh.helm.release.v1.fleet.v3 helm.sh/release.v1 1 40h sh.helm.release.v1.fleet.v4 helm.sh/release.v1 1 40h stv-aggregation Opaque 3 43h Any help would be great appreciated

If anybody is managing deployments that contain Kubernetes Jobs with CD tools like Fleet, this blog post may be of interest: https://www.suse.com/c/rancher_blog/rancher-fleet-tips-for-kubernetes-jobs-deployment-strategies-in-continuous-delivery-scenarios/

Hi, I have a k3s cluster that is managed by Rancher and the fleet-agent deployment to the k3s cluster is behaving very strange. It's almost as if I have two deployments of the fleet-agent competing with each other, I can see with

helm list

that

REVISION

is increased a couple of times and then the helm chart for fleet-agent is removed and it all repeats again. The cycle time of installing fleet-agent with helm and then bumping

REVISION

and uninstalling the helm chart for fleet-agent is ~10 seconds. It should be said that we have used this cluster for upgrade-tests earlier that might have caused this to happen, but if I can get some hints where I might find the cause for this I will be very grateful since I have spent too much time going round in circles. Note: 1. The fleet-agent pod is not restarted, but the fleet-agent bundle is jumping between states "Pending" and "Wait applied". 2. We are managing 12 clusters with this Rancher installation and this k3s cluster is the only one fighting the fleet-agent bundle.

fleet-agent time="2022-10-12T14:38:44Z" level=error msg="error syncing 'cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd': handler bundle-deploy: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a70272d3-e2fb-4864-aa9f-a7565c7570b4, UID in object meta: , handler bundle-monitor: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a70272d3-e2fb-4864-aa9f-a7565c7570b4, UID in object meta: , requeuing"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="Helm: Deleting release fleet-agent-k3s-rnd 1"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="Helm: Installing fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="Helm: Deleting release fleet-agent-k3s-rnd 1"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="Helm: Installing fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=error msg="error syncing 'cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd': handler bundle-deploy: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aae9c7b4-452f-418a-a5bc-4a43b6f2fa66, UID in object meta: , handler bundle-monitor: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aae9c7b4-452f-418a-a5bc-4a43b6f2fa66, UID in object meta: , requeuing"

When looking at the gitrepo it would appear as though the targetCustomization isn't working properly. The web UI view suggests there are resources created for the wrong environments via the clusterSelector. I've checked the master cluster for these additional resources that shouldn't be there and I can confirm they weren't created. I can also confirm that from a helm perspective all seems to be working properly. When retrieving the values file from the helm bundle i get back only the values meant for the TEST environment. If i was using the below in fleet.yaml then looking at rancher web UI would suggest that all 3 values sections were merged/used at once when only TEST used be used. My clusters name is called local. It has a label "env=test".

targetCustomizations:
- name: dev
  helm:
    values:
      replication: false
  clusterSelector:
    matchLabels:
      env: dev

- name: test
  helm:
    values:
      replicas: 3
  clusterSelector:
    matchLabels:
      env: test

- name: prod
  helm:
    values:
      serviceType: LoadBalancer
      replicas: 3
  clusterSelector:
    matchLabels:
      env: prod

If, say, due to operator error, a Manager Initiated Fleet agent Helm Release (and its values) were to have been deleted, is there an easy way to reinstall a release of the agent by the Manager or should I do the "Agent Initiated" flow to restore the agent?

Hi All 🙂 I've been playing around with a fleet.yaml, cause we have a single git repo that deploys to 2 different env's with their own customizations. But for some reason it doesn't seem to work correctly. We have a 'default' values.yaml and a values-ENV.yaml. For example:

targetCustomizations:
  - name: distinct
    helm:
      valuesFiles:
        - values-DISTINCT.yaml
    clusterSelector:
      matchLabels:
        env: distinct

But its not deploying the things from values.yaml (which should be used as 'base')

Seems dead here 😄

https://forums.rancher.com/t/fleet-values-in-external-file/19048 seems like it still doesn't work 😞

Is there a way to add notifications for when a deployment occurs and receive its state continuously when it changes? Something like https://github.com/rancher/fleet/issues/317

was there any follow up on this? In my case using a 2.6.8 version installed from scratch, I see the fleet-agent-local stuck in Pending-Upgrade, but not the Uninstalling

Hey y’all, cross posting my thread from #general because I’ve determined this appears to be fleet related https://rancher-users.slack.com/archives/C3ASABBD1/p1667334699205249

Hello, everyone. I am using fleet, everything is working correctly. Mistakenly though, a statefull object was deleted via Rancher and now fleet is in "modified" state on that bundle element, even though I recreated it with different name. Do you have any suggestions?

Any idea if it's possible to use fleet-local and add an extra values.yaml except the default one?

Hello, everyone! When i add two Ingress, fleet return this message:

Modified(1) [Bundle ci-k8s-aon-oneflex2]; <http://certificate.cert-manager.io|certificate.cert-manager.io> <namespace>/<secret name>-secret extra; <http://certificate.cert-manager.io|certificate.cert-manager.io> <namespace>/<secret name>-secret extra

Any idea to resolve?

Hi everyone!

Has anyone been successful in getting sop (or other 2-way encryption tool) working with fleet?

Hi everyone. Fleet v0.5.0 is out https://github.com/rancher/fleet/releases/tag/v0.5.0 • Add support for OCI registry based charts in #975 • Adds priorityClassName to deployments in #983 • Support cluster specific registry in #984, #1060 • Add cluster labels to agent config creation and a couple of bug fixes • Fix long bundle names in #964 • Remove ignored fields from status struct, to avoid loop in #990 • Fix for cluster registration token performance degradation in #998 • Fix authentication for OCI images in #998 • Stop requeuing on failed deployments in #1045 congrats to the team for the hard work!

Hello We just updated to 2.6.9 and are seeing some issues with helm authentication in fleet. Getting 401 errors when authentication with known good passwords to multiple repos. Configuring the repos to not require auth means we get access. Any ideas?

New error for my helm auth issue: Time="2022-11-16T081211Z" level=info msg="updated: fleet-default/password-file-env-name" time="2022-11-16T081211Z" level=fatal msg="unable to parse requirement: values[0][fleet.cattle.io/repo-name]: Invalid value: \"--password-file\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')"

hi i have a fleet yaml that is instaling in metallb. i have a diff in the yaml and it is ignoring the correct resources. But fleet is coming back with the error

time="2022-11-25T14:11:46Z" level=fatal msg="error converting YAML to JSON: yaml: line 31: found character that cannot start any token"

The fleet.yaml looks like this:

defaultnamespace: metallb-system

helm: # Use a custom location for the Helm chart. This can refer to any go-getter URL or # OCI registry based helm chart URL e.g. "oci://ghcr.io/fleetrepoci/guestbook". # This allows one to download charts from most any location. Also know that # go-getter URL supports adding a digest to validate the download. If repo # is set below this field is the name of the chart to lookup chart: metallb # A https URL to a Helm repo to download the chart from. It's typically easier # to just use

chart

field and refer to a tgz file. If repo is used the # value of

chart

will be used as the chart name to lookup in the Helm repository. repo: https://charts.bitnami.com/bitnami # A custom release name to deploy the chart as. If not specified a release name # will be generated. releaseName: metallb-bitnami # The version of the chart or semver constraint of the chart to find. If a constraint # is specified it is evaluated each time git changes. # The version also determines which chart to download from OCI registries. version: 4.1.11 values: speaker: secretName: metallb-bitnami-memberlist secretKey: secretkey # Any values that should be placed in the

values.yaml

and passed to helm during # install. # Override immutable resources. This could be dangerous. force: false # Set the Helm --atomic flag when upgrading diff: comparePatches: - apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: addresspools.metallb.io operations: - {"op":"remove", "path":"/spec/conversion/webhook/clientConfig/caBundle"} - {"op":"remove", "path":"/status/acceptedNames/kind"} - {"op":"remove", "path":"/status/acceptedNames/plural"} - {"op":"remove", "path":"/status/conditions"} - {"op":"remove", "path":"/status/storedVersions"} - apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: bgppeers.metallb.io operations: - {"op":"remove", "path":"/spec/conversion/webhook/clientConfig/caBundle"} - {"op":"remove", "path":"/status/acceptedNames/kind"} - {"op":"remove", "path":"/status/acceptedNames/plural"} - {"op":"remove", "path":"/status/conditions"} - {"op":"remove", "path":"/status/storedVersions"} clusterGroup: downstream any help appreciated

HI, Im deploying rke2 clusters with cluster-templates and fleet, and realized that somehow rancher changes something in “machinePoolConfig” after deploy. Is there a way to define a jsonPointer patch for all elements ? some kind of wildcard ?

"/spec/rkeConfig/machinePools/0/etcdRole"
"/spec/rkeConfig/machinePools/1/etcdRole"
"/spec/rkeConfig/machinePools/2/etcdRole"

the real “problem” ist that, i specifically set “etcdRole = False” in my Template and it looks like that all non-default elements get deleted in the config after deployment

we are running rancher 2.6.9 and that is fleet 0.4.0 and we are trying to deploy helm charts via fleet. We have a web server (nexus sonatype repository ) that is hosting the tgz files for the helm chart ....traefik. We are getting errors talking about x509 certificate error because the cert on the web server has a custom CA. In the README it gives instruction on how to add secrets. We added the CAcert as a secret create secret -n fleet-default generic helm --from-file="\path to cert\cacerts" , and added the helmSecret directive in the gitrepo object under spec. However I still get the error x509 error. I tried naming the file cacerts, cacert.pem , but fleet still doesnt accept the certificate and therefore does not pull the file. Any help will be appreciated.

is there a way to specify a order of objects when they are created ? for example i have a namespace, role, serviceaccount objects and the creation fails because of “namespace does not exist”. “default: Namespace” is OK. But I’d like to annotate the namespace as well