Has anyone ever seen fleet agent pods failing to start with

/usr/lib/ca-certificates/update.d/99certbundle.run: line 21: /var/lib/ca-certificates/ca-bundle.pem.new: Read-only file system

?

Hi All, not sure if this goes here or in another channel. We recently upgraded our DR rancher instance to 2.6.9 from 2.6.3, We have one Windows enabled cluster. Everything is working except the fleet-agent. It is stuck in "creatingcontainer." When I describe the pod, I see this error:

Warning  FailedMount  45s   kubelet            MountVolume.SetUp failed for volume "kube-api-access-k5tht" : chown c:\var\lib\kubelet\pods\6c04872e-2cf8-468c-93ad-a03574b7c9ea\volumes\<http://kubernetes.io|kubernetes.io>~projected\kube-api-access-k5tht\..2022_12_07_15_29_15.258699061\token: not supported by windows

Did something change with Fleet that Rancher doesn't update properly?

Hello all. I'm standing up a new Rancher cluster using version 2.6.9 on Kubernetes 1.23.12 in Azure AKS. I have a git repo set up with Fleet deployments that reference a self-hosted chart museum instance. I am having trouble getting Rancher to properly authenticate to chart museum using basic auth. I keep getting 401 errors. When I check the request from the chart museum side, I don't see an auth header being set. I've tried setting the Helm Authentication to an existing Kubernetes secret (which says not found) and I've also tried creating a new secret via the UI. Using the same creds, I'm able to download the chart in my browser. Any help is much appreciated

I posted this in #general, but I might have better luck in here We tried to move the local cluster from

fleet-local

to

fleet-defaul

by following the instructions here: https://fleet.rancher.io/troubleshooting#migrate-the-local-cluster-to-the-fleet-default-cluster After we initialized the move, I could see this in the logs:

rancher-5677f59677-shbs7 rancher 2022/12/07 12:39:42 [ERROR] error syncing 'local': handler provisioning-cluster-create: failed to create fleet-default/local <http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>, Kind=
Cluster for provisioning-cluster-create local: admission webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>" denied the request: cluster name must be 63 characters or fewer, cannot be "local" nor of the form "c
-xxxxx", requeuing

While this was happening, the local cluster had been removed from the

fleet-local

workspace but for the reason above, it did not show up in

fleet-default

. We managed to get the cluster back to

fleet-local

again after editing the object, but it would have been nice to have all the "Continuous delivery" clusters in the same workspace. If you have any hints how to make that migration successful, please share 🙂

Hi, I have a bundle with multiple kustomize variants: • qa • stage • prod If I have 3 clusters with matching labels everything happens as expected. But how about if I want to have only 2 clusters: • qa and stage • prod the target customizations in the bundle already have namespaces, so "should work" and I would expect 2 helm releases on the shared cluster But only one happens. I think it boils down, that for the rule that: for one Bundle you can have only 1 BundeDeployment per cluster.

How can I modify the example: https://github.com/rancher/fleet-examples/blob/master/multi-cluster/kustomize/fleet.yaml so that the first 2 environment is deployed to a shared cluster? do I have to create a new Bundle? that seems just to much copy/paste for what I try to achive

Hi @here, Glad to join the fleet channel. My question: How to set the Rancher project by name for a namespace using fleet? I used this yaml and it works:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    <http://field.cattle.io/projectId|field.cattle.io/projectId>: c-pkz94:p-5n2wt
  name: c-providers
spec:
  finalizers:
  - kubernetes

But the projectId (ie`c-pkz94:p-5n2wt`) isn't consistent between rancher cluster.

Hi, How to set the kustomize parameter --load-restrictor to LoadRestrictionsNone when fleet process a bundle?

I’m setting up an Helm chart to be deployed using Fleet the deployed app should be configured “Kubernetes-cluster-display-name”. In my values file i’m passing the following: targetName: “{{ management.cattle.io/cluster-display-name }}” The cluster name is not displaying as intended, any help on how to injecting a variable into a helm values?

@square-telephone-53524

<http://global.fleet.clusterLabels.management.cattle.io/cluster-display-name|global.fleet.clusterLabels.management.cattle.io/cluster-display-name>

without quotes or braces

@mammoth-postman-10874 thank you for the feedback, i’ve updated my fleet and values file. Testing now

@mammoth-postman-10874 - this worked, thank you very much

I’m trying to deploy some basic resources to my downstream clusters using Fleet and going the Kustomize route https://github.com/rancher/fleet-examples/tree/master/multi-cluster/kustomize. Works fine but i also want to add some helm charts to deployed and can’t find any docs on how to? Found some refrences to adding

helmChartInflationGenerator

or

helmCharts

to the

kustomization.yml

but they dont seem to get picked up

Hi @here, What is your recommended layout for fleet GitRepo? • have a GitRepo for each bundle, even if they are in the same git repository?

- bundles/
   - bundle 1            <-- GitRepo
     fleet.yaml
   - bundle 2            <-- GitRepo
     fleet.yaml

• have a single GitRepo for a git repository, with many bundles under?

- bundles/               <-- GitRepo
   - bundle 1
     fleet.yaml
   - bundle 2
     fleet.yaml

I use the second depending on the use case. So for example we have base infrastructure apps like traefik, long horn and metallb. So we keep those in one group. Other type of application apps we keep in another repo

Hi there, Happy New Year What is the correctest way to not install fleet and fleet-controller ? (using the standard rancher helm chart, 2.7.0) Thank you so much

@mysterious-lock-12214 you can’t rancher-agents are deployed by fleet

hey, any idea if I can leverage helm's post-render option with fleet? The reason for that is because I want to override a 3rd party template coming from helm but I don't want to fork and fix it. I explained this in rancher's forum if you are interested in more details

is there a way to integrate fleet into the rbac functions of rancher? I saw when I use AD authentcation there is no option to access fleet. did I oversee something?

Hello, I'm trying to bootstrap ArgoCD with Fleet without any success. This is the error message I get:

Unable to build kubernetes objects from release manifest: resource mapping not found for name: "my-demo" namespace: "system" from "": no matches for kind "Application" in version "<http://argoproj.io/v1alpha1|argoproj.io/v1alpha1>" ensure CRDs are installed first

Here's how my repo looks like:

├── Chart.lock

├── charts

│   ├── argo-cd-5.13.7.tgz

│   └── ingress-nginx-4.1.1.tgz

├── Chart.yaml

├── fleet.yaml

├── templates

│   └── app-of-apps.yaml

└── values.yaml

fleet.yaml file:

defaultNamespace: argocd

targetCustomizations:

- name: qa

helm:

values:

replication: false

clusterGroup: qa-clusters

Chart.yaml file:

apiVersion: v2

name: system-chart

description: A wrapper chart for argo CD

type: application

version: 0.8.5

dependencies:

- name: argo-cd

version: 5.13.7

repository: <https://argoproj.github.io/argo-helm>

condition: argo-cd.enabled

- name: ingress-nginx

version: 4.1.1

repository: <https://kubernetes.github.io/ingress-nginx>

condition: ingress-nginx.enabled

As mentioned on this page https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet#helm-chart-dependencies I did a helm dependencies build $chart to cover dependencis and add to my repo. Can someone help me or give me some guidance?

Hi fleeters! I accidently deployed another gitrepo to the same gitrepo name and am now trying to fix things. I deleted the bundle, gitrepo, namespace and then tried to redploy the gitrepo, which results in:

rendered manifests contain a resource that already exists. Unable to continue with install: Namespace "foo" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>": must be set to "Helm"; annotation validation error: missing key "<http://meta.helm.sh/release-name|meta.helm.sh/release-name>": must be set to "admin-foo"; annotation validation error: missing key "<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>": must be set to "foo"

Any ideas what is going on?

@victorious-analyst-3332 has left the channel

How do you prevent a change in a helm template to be deployed to one cluster (say, production) but make sure it will be deployed to another (say, stage)?

Do you use

helm test

in your CI? How does your setup look like? I'm curious to see how other's approach this if at all - especially how you integrate it with fleet at the end (there aren't many examples out there) 🙏

Hello, we uses fleet to deploy many helm chart. We have trouble with Keda Helm Chart. When we try to deploy we have this error in fleet "hart requires kubeVersion: >=v1.23.0-0 which is incompatible with Kubernetes v1.20.0" but we are on RKE2 1.24.9 on all our clusters. Anyone have a solution or an explanation (github issue for example)?

@orange-airplane-98016 has left the channel

does anyone know what

managedChart

is doing ? i got one cluster deployed with fleet stuck at

ManagedChart generation is 2, but latest observed generation is

and I wonder how i could trigger the upgrade to generation 2. alsothe helm release

rancher-monitoring-crd

seems to be installed corectly on the downstream cluster

Hi all, I am trying to follow the installation here: https://fleet.rancher.io/quickstart, on Mac Rancher desktop. Although crd installation is done properly, but when I try to install fleet :

helm -n cattle-fleet-system install --create-namespace --wait \ fleet <https://github.com/rancher/fleet/releases/download/v0.5.0/fleet-0.5.0.tgz>

I get following error:

Error: INSTALLATION FAILED: timed out waiting for the condition

Any suggestions, please? Thank you. Expected behaviour To properly install fleet on Rancher Desktop. Steps To Reproduce 1. Install fleet using :

Error: INSTALLATION FAILED: timed out waiting for the condition

2. Error:

Error: INSTALLATION FAILED: timed out waiting for the condition

Environment

- Mac
- Rancher Desktop

Logs: Running the install command with

--debug

getting following output:

install.go:192: [debug] Original chart version: ""
install.go:209: [debug] CHART PATH: /Downloads/fleet-0.5.1.tgz
client.go:229: [debug] checking 17 resources for changes
client.go:512: [debug] Looks like there are no changes for ServiceAccount "gitjob"
client.go:512: [debug] Looks like there are no changes for ServiceAccount "fleet-controller"
client.go:512: [debug] Looks like there are no changes for ServiceAccount "fleet-controller-bootstrap"
client.go:512: [debug] Looks like there are no changes for ConfigMap "fleet-controller"
client.go:512: [debug] Looks like there are no changes for ClusterRole "gitjob"
client.go:512: [debug] Looks like there are no changes for ClusterRole "fleet-controller"
client.go:512: [debug] Looks like there are no changes for ClusterRole "fleet-controller-bootstrap"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "gitjob-binding"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "fleet-controller"
client.go:512: [debug] Looks like there are no changes for ClusterRoleBinding "fleet-controller-bootstrap"
client.go:512: [debug] Looks like there are no changes for Role "gitjob"
client.go:512: [debug] Looks like there are no changes for Role "fleet-controller"
client.go:512: [debug] Looks like there are no changes for RoleBinding "gitjob"
client.go:512: [debug] Looks like there are no changes for RoleBinding "fleet-controller"
client.go:512: [debug] Looks like there are no changes for Service "gitjob"
client.go:512: [debug] Looks like there are no changes for Deployment "gitjob"
client.go:512: [debug] Looks like there are no changes for Deployment "fleet-controller"
wait.go:66: [debug] beginning wait for 17 resources with timeout of 5m0s
ready.go:277: [debug] Deployment is not ready: cattle-fleet-system/gitjob. 0 out of 1 expected pods are ready
ready.go:277: [debug] Deployment is not ready: cattle-fleet-system/gitjob. 0 out of 1 expected pods are ready
...
ready.go:277: [debug] Deployment is not ready: cattle-fleet-system/gitjob. 0 out of 1 expected pods are ready
ready.go:277: [debug] Deployment is not ready: cattle-fleet-system/gitjob. 0 out of 1 expected pods are ready
Error: INSTALLATION FAILED: timed out waiting for the condition
helm.go:84: [debug] timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
<http://helm.sh/helm/v3/cmd/helm/install.go:141|helm.sh/helm/v3/cmd/helm/install.go:141>
<http://github.com/spf13/cobra.(*Command).execute|github.com/spf13/cobra.(*Command).execute>
<http://github.com/spf13/cobra@v1.5.0/command.go:872|github.com/spf13/cobra@v1.5.0/command.go:872>
<http://github.com/spf13/cobra.(*Command).ExecuteC|github.com/spf13/cobra.(*Command).ExecuteC>
<http://github.com/spf13/cobra@v1.5.0/command.go:990|github.com/spf13/cobra@v1.5.0/command.go:990>
<http://github.com/spf13/cobra.(*Command).Execute|github.com/spf13/cobra.(*Command).Execute>
<http://github.com/spf13/cobra@v1.5.0/command.go:918|github.com/spf13/cobra@v1.5.0/command.go:918>
main.main
<http://helm.sh/helm/v3/cmd/helm/helm.go:83|helm.sh/helm/v3/cmd/helm/helm.go:83>
runtime.main
runtime/proc.go:250
runtime.goexit
runtime/asm_amd64.s:1571

Continuing with the above issue, I see that when I descripe in the POD:

Failed to pull image "rancher/gitjob:v0.1.32-security1": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/gitjob:v0.1.32-security1|docker.io/rancher/gitjob:v0.1.32-security1>": failed to resolve reference "<http://docker.io/rancher/gitjob:v0.1.32-security1|docker.io/rancher/gitjob:v0.1.32-security1>": pulling from host <http://registry-1.docker.io|registry-1.docker.io> failed with status code [manifests v0.1.32-security1]: 403 Forbidden