https://rancher.com/ logo
Join Slack
Powered by
Hi all, Since a few days we've noticed that our `...
# general
k
Hi all, Since a few days we've noticed that our 
fleet-agent
is constant being restarted. It could be well possible this is caused by either a Rancher update from 2.11 to 2.12 or caused by switching a root certificate from self-signed to a provided root certificate (following these docs). In the 
fleet-controller/fleet-agentmanagement
we see the following logs constantly
Copy code
time="2025-11-13T10:44:39Z" level=info msg="Deleted old agent for cluster (fleet-local/local) in namespace cattle-fleet-local-system"
time="2025-11-13T10:44:39Z" level=info msg="Cluster import for 'fleet-local/local'. Deployed new agent"
time="2025-11-13T10:45:00Z" level=info msg="Waiting for service account token key to be populated for secret cluster-fleet-local-local-1a3d67d0a899/request-cs9x7-8645b8de-5e30-4eb0-a9fe-dc96f1081856-token"
time="2025-11-13T10:45:02Z" level=info msg="Cluster registration request 'fleet-local/request-cs9x7' granted, creating cluster, request service account, registration secret"
The fleet-agent in the cattle-fleet-local-system isn't reporting any errors but just reposts
Copy code
I1113 10:44:40.589439       1 leaderelection.go:257] attempting to acquire leader lease cattle-fleet-local-system/fleet-agent...
{"level":"info","ts":"2025-11-13T10:44:40Z","logger":"setup","msg":"new leader","identity":"fleet-agent-6d5f55c7d7-4pncc-1"}
I1113 10:45:00.267179       1 leaderelection.go:271] successfully acquired lease cattle-fleet-local-system/fleet-agent
{"level":"info","ts":"2025-11-13T10:45:00Z","logger":"setup","msg":"renewed leader","identity":"fleet-agent-5cf8799b4c-xn274-1"}
time="2025-11-13T10:45:00Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2025-11-13T10:45:00Z" level=info msg="Creating clusterregistration with id 'pwvp47nf7r8pg8zfmd4tx7vxb6rhr5dwv2gcnn2m6zlrtmt54ss9kl' for new token"
time="2025-11-13T10:45:02Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-9072b2e8eac3a21368e0428adc1a0244a61acd4ee571c7f88f574d905cd52' on management cluster for request 'fleet-local/request-cs9x7': secrets \"c-9072b2e8eac3a21368e0428adc1a0244a61acd4ee571c7f88f574d905cd52\" not found"
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"setup","msg":"successfully registered with upstream cluster","namespace":"cluster-fleet-local-local-1a3d67d0a899"}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"setup","msg":"listening for changes on upstream cluster","cluster":"local","namespace":"cluster-fleet-local-local-1a3d67d0a899"}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"setup","msg":"Starting controller","metricsAddr":":8080","probeAddr":":8081","systemNamespace":"cattle-fleet-local-system"}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"starting server","name":"health probe","addr":"0.0.0.0:8081"}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"Starting EventSource","controller":"bundledeployment","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"BundleDeployment","source":"kind source: *v1alpha1.BundleDeployment"}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"setup","msg":"Starting cluster status ticker","checkin interval":"15m0s","cluster namespace":"fleet-local","cluster name":"local"}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"Starting EventSource","controller":"drift-reconciler","source":"channel source: 0xc00078f3b0"}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"Starting Controller","controller":"drift-reconciler"}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"Starting workers","controller":"drift-reconciler","worker count":50}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"Starting Controller","controller":"bundledeployment","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"BundleDeployment"}
{"level":"info","ts":"2025-11-13T10:45:04Z","msg":"Starting workers","controller":"bundledeployment","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"BundleDeployment","worker count":50}
{"level":"info","ts":"2025-11-13T10:45:04Z","logger":"bundledeployment.helm-deployer.install","msg":"Upgrading helm release","controller":"bundledeployment","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"BundleDeployment","BundleDeployment":{"name":"fleet-agent-local","namespace":"cluster-fleet-local-local-1a3d67d0a899"},"namespace":"cluster-fleet-local-local-1a3d67d0a899","name":"fleet-agent-local","reconcileID":"1e4df644-069b-4d05-84ed-2c447bc54d15","commit":"","dryRun":false}
{"level":"info","ts":"2025-11-13T10:45:05Z","logger":"bundledeployment.deploy-bundle","msg":"Deployed bundle","controller":"bundledeployment","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"BundleDeployment","BundleDeployment":{"name":"fleet-agent-local","namespace":"cluster-fleet-local-local-1a3d67d0a899"},"namespace":"cluster-fleet-local-local-1a3d67d0a899","name":"fleet-agent-local","reconcileID":"1e4df644-069b-4d05-84ed-2c447bc54d15","deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","appliedDeploymentID":"s-c1fc5eeb18677acb8c4a8fd2054c2c40c4022f002ea06437f9b108731be8f:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","release":"cattle-fleet-local-system/fleet-agent-local:20","DeploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623"
And afterwards the fleet-agent is restarted again... Any idea's on what could be wrong and how to fix it? We've already redeployed the fleet-controller, fleet-agent deployments, reinstalled the fleet-agent and fleet-controller helm charts with no luck
b
Not at all what you're asking, but you mentioned upgrading from 2.11 to 2.12. When I did that, I didn't change the global variables ui-index and ui-dashboard-index to point to release-2.12, which caused me issues (though not immediately). I take it you've covered this already?
k
Where did you set those variables?
Ah found them, no the ui-index and ui-dashboard-index both point tho 2.12
👍 1
c
What is the restart reason? Is it exiting? Is it being oom killed? Is it being deleted and recreated? Show the pod yaml and/or describe the pod.
k
I couldn’t find a reason the fleet agents gets recreated. It think the controller recreates it after 3min.
The attached logs from the agent are the last logs. No Kubernetes events stating any OOMkilled. Just that the pod is stopped
c
Look at pod status and events, not logs. Unless the process in the pod is exiting with an error the pod logs will tell you nothing.
k
I’ll give it a look (back at my laptop in an hour)
These are the events sorted on timestamp
Copy code
cattle-fleet-local-system   85s         Normal   SuccessfulCreate    replicaset/fleet-agent-679fc4658c   Created pod: fleet-agent-679fc4658c-4xqsx
cattle-fleet-local-system   85s         Normal   Killing             pod/fleet-agent-5f79d68fcf-j89d6    Stopping container fleet-agent
cattle-fleet-local-system   85s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-679fc4658c from 0 to 1
cattle-fleet-local-system   85s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-679fc4658c from 0 to 1
cattle-fleet-local-system   84s         Normal   Created             pod/fleet-agent-679fc4658c-4xqsx    Created container: fleet-agent
cattle-fleet-local-system   84s         Normal   Started             pod/fleet-agent-679fc4658c-4xqsx    Started container fleet-agent
cattle-fleet-local-system   84s         Normal   Pulled              pod/fleet-agent-679fc4658c-4xqsx    Container image "rancher/fleet-agent:v0.13.1" already present on machine
cattle-fleet-local-system   72s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-5f79d68fcf from 0 to 1
cattle-fleet-local-system   72s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-5f79d68fcf from 0 to 1
cattle-fleet-local-system   72s         Normal   Killing             pod/fleet-agent-679fc4658c-4xqsx    Stopping container fleet-agent
cattle-fleet-local-system   72s         Normal   SuccessfulCreate    replicaset/fleet-agent-5f79d68fcf   Created pod: fleet-agent-5f79d68fcf-wd25c
cattle-fleet-local-system   72s         Normal   SuccessfulCreate    replicaset/fleet-agent-5f79d68fcf   Created pod: fleet-agent-5f79d68fcf-bjr86
cattle-fleet-local-system   71s         Normal   Pulled              pod/fleet-agent-5f79d68fcf-wd25c    Container image "rancher/fleet-agent:v0.13.1" already present on machine
cattle-fleet-local-system   71s         Normal   Created             pod/fleet-agent-5f79d68fcf-wd25c    Created container: fleet-agent
cattle-fleet-local-system   71s         Normal   Started             pod/fleet-agent-5f79d68fcf-wd25c    Started container fleet-agent
cattle-fleet-local-system   62s         Normal   Pulled              pod/fleet-agent-679fc4658c-ttnzw    Container image "rancher/fleet-agent:v0.13.1" already present on machine
cattle-fleet-local-system   62s         Normal   SuccessfulCreate    replicaset/fleet-agent-679fc4658c   Created pod: fleet-agent-679fc4658c-9bcq8
cattle-fleet-local-system   62s         Normal   SuccessfulCreate    replicaset/fleet-agent-679fc4658c   Created pod: fleet-agent-679fc4658c-ttnzw
cattle-fleet-local-system   62s         Normal   Killing             pod/fleet-agent-5f79d68fcf-wd25c    Stopping container fleet-agent
cattle-fleet-local-system   62s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-679fc4658c from 0 to 1
cattle-fleet-local-system   62s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-679fc4658c from 0 to 1
cattle-fleet-local-system   61s         Normal   Created             pod/fleet-agent-679fc4658c-ttnzw    Created container: fleet-agent
cattle-fleet-local-system   61s         Normal   Started             pod/fleet-agent-679fc4658c-ttnzw    Started container fleet-agent
cattle-fleet-local-system   21s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-5f79d68fcf from 0 to 1
cattle-fleet-local-system   21s         Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-5f79d68fcf from 0 to 1
cattle-fleet-local-system   21s         Normal   Killing             pod/fleet-agent-679fc4658c-ttnzw    Stopping container fleet-agent
cattle-fleet-local-system   21s         Normal   SuccessfulCreate    replicaset/fleet-agent-5f79d68fcf   Created pod: fleet-agent-5f79d68fcf-qpz7t
cattle-fleet-local-system   21s         Normal   SuccessfulCreate    replicaset/fleet-agent-5f79d68fcf   Created pod: fleet-agent-5f79d68fcf-dqr6h
cattle-fleet-local-system   20s         Normal   Created             pod/fleet-agent-5f79d68fcf-dqr6h    Created container: fleet-agent
cattle-fleet-local-system   20s         Normal   Pulled              pod/fleet-agent-5f79d68fcf-dqr6h    Container image "rancher/fleet-agent:v0.13.1" already present on machine
cattle-fleet-local-system   20s         Normal   Started             pod/fleet-agent-5f79d68fcf-dqr6h    Started container fleet-agent
cattle-fleet-local-system   1s          Normal   Killing             pod/fleet-agent-5f79d68fcf-dqr6h    Stopping container fleet-agent
cattle-fleet-local-system   1s          Normal   SuccessfulCreate    replicaset/fleet-agent-679fc4658c   Created pod: fleet-agent-679fc4658c-kk8w5
cattle-fleet-local-system   1s          Normal   SuccessfulCreate    replicaset/fleet-agent-679fc4658c   Created pod: fleet-agent-679fc4658c-zzxzn
cattle-fleet-local-system   1s          Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-679fc4658c from 0 to 1
cattle-fleet-local-system   1s          Normal   ScalingReplicaSet   deployment/fleet-agent              Scaled up replica set fleet-agent-679fc4658c from 0 to 1
It feels like two ScalingReplicaSet are battling to deploy a fleet-agent. But when I inspect the deployments and replicasets I only have one fleet-agent configured.
c
what’s going on with the fleet-agent deployment? what is modifying or scaling that?
you keep getting new replicasets created for that deployment, which in turn creates new pods as they get scaled up/down. So you need to figure out what is changing that deployment.
this is just normal behavior when a deployment is changed
k
Ill also get constantly new deployments. It looks like the fleet-controller is responsible for creating those.
Isn't their a way to completely reset fleet in our Rancher management cluster?
But this is what I see in the fleet-controller, and those times matches the moments of the restarts of my fleet-agent
Copy code
time="2025-11-13T17:45:33Z" level=info msg="Cluster import for 'fleet-local/local'. Deployed new agent"
time="2025-11-13T17:45:43Z" level=info msg="Update agent bundle for cluster fleet-local/local"
time="2025-11-13T17:45:43Z" level=info msg="Deleted old agent for cluster (fleet-local/local) in namespace cattle-fleet-local-system"
But I can't seem to figure out why the fleet-controller constantly keeps importing, updating and deleting the agent
The cluster.fleet.cattle.io states in its status 
WaitApplied(1) [Bundle fleet-agent-local]
and it status.lastSeen stays 
null
, while other clusters have a valid timestamp.
But it looks like it is constantly switching between;. When watching the resource I get this output
Copy code
kubectl get <http://clusters.fleet.cattle.io|clusters.fleet.cattle.io> -n fleet-local local --watch -o wide
NAME    BUNDLES-READY   LAST-SEEN              STATUS
local   1/1             2025-11-13T18:16:53Z
local   1/1             2025-11-13T18:16:53Z
local   1/1             2025-11-13T18:16:53Z
local   0/1             2025-11-13T18:16:53Z   WaitApplied(1) [Bundle fleet-agent-local]
local   0/1             <no value>             WaitApplied(1) [Bundle fleet-agent-local]
local   0/1             <no value>             WaitApplied(1) [Bundle fleet-agent-local]
local   1/1             <no value>
local   1/1             2025-11-13T18:19:54Z
local   1/1             2025-11-13T18:19:54Z
local   1/1             2025-11-13T18:19:54Z
local   1/1             2025-11-13T18:19:54Z
local   0/1             2025-11-13T18:19:54Z   WaitApplied(1) [Bundle fleet-agent-local]
local   0/1             <no value>             WaitApplied(1) [Bundle fleet-agent-local]
local   0/1             <no value>             WaitApplied(1) [Bundle fleet-agent-local]
c
What is getting changed on it?
k
That’s something I can’t seem to figure out. Could it be possible that the fleet-agent once it’s ready registers itself somehow at the fleet-controller? And if that’s not happening it restarts the agent? We recently updated the private CA of Rancher, and I’m not sure if that could cause this issue.
I've also noticed that in the rancher pod logs, the following error is logged 
http: TLS handshake error from 10.36.4.254:36930: remote error: tls: bad certificate
. The IP mentioned in the log is the IP of the fleet-controller..
c
does the fleet-controller pod log have anything interesting while this is all going on?
k
No not really as far as I understand. Just the logs I already shared in the initial post
Maybe only interesting is that the controller (fleet-agentmanager) keeps logging 
time="2025-11-13T20:52:14Z" level=info msg="Deleted old agent for cluster (fleet-local/local) in namespace cattle-fleet-local-system"
.
And the fleet-controller logs
Copy code
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"error","ts":"2025-11-13T20:53:53Z","msg":"Reconciler error","controller":"bundle","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"Bundle","Bundle":{"name":"fleet-agent-local","namespace":"fleet-local"},"namespace":"fleet-local","name":"fleet-agent-local","reconcileID":"e67a2d76-f371-4c08-9dad-81ca036093d5","error":"failed to create bundle deployment: failed to get content resource: <http://Content.fleet.cattle.io|Content.fleet.cattle.io> \"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9\" not found","stacktrace":"<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler>\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"bundle","msg":"metadata.finalizers: \"fleet-agent-local\": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers","controller":"bundle","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"Bundle","Bundle":{"name":"fleet-agent-local","namespace":"fleet-local"},"namespace":"fleet-local","name":"fleet-agent-local","reconcileID":"02491bca-b77c-4f0e-8761-046bf6ecc232"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"bundle","msg":"Updated bundledeployment","controller":"bundle","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"Bundle","Bundle":{"name":"fleet-agent-local","namespace":"fleet-local"},"namespace":"fleet-local","name":"fleet-agent-local","reconcileID":"02491bca-b77c-4f0e-8761-046bf6ecc232","manifestID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9","bundledeployment":{"metadata":{"name":"fleet-agent-local","namespace":"cluster-fleet-local-local-1a3d67d0a899","creationTimestamp":null,"labels":{"<http://fleet.cattle.io/bundle-name|fleet.cattle.io/bundle-name>":"fleet-agent-local","<http://fleet.cattle.io/bundle-namespace|fleet.cattle.io/bundle-namespace>":"fleet-local","<http://fleet.cattle.io/cluster|fleet.cattle.io/cluster>":"local","<http://fleet.cattle.io/cluster-namespace|fleet.cattle.io/cluster-namespace>":"fleet-local","<http://fleet.cattle.io/managed|fleet.cattle.io/managed>":"true"},"finalizers":["<http://fleet.cattle.io/bundle-deployment-finalizer|fleet.cattle.io/bundle-deployment-finalizer>"]},"spec":{"stagedOptions":{"defaultNamespace":"cattle-fleet-local-system","helm":{"takeOwnership":true}},"stagedDeploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","options":{"defaultNamespace":"cattle-fleet-local-system","helm":{"takeOwnership":true}},"deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623"},"status":{"display":{},"resourceCounts":{"ready":0,"desiredReady":0,"waitApplied":0,"modified":0,"orphaned":0,"missing":0,"unknown":0,"notReady":0}}},"deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","operation":"updated"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"bundle","msg":"Unchanged bundledeployment","controller":"bundle","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"Bundle","Bundle":{"name":"fleet-agent-local","namespace":"fleet-local"},"namespace":"fleet-local","name":"fleet-agent-local","reconcileID":"5bcab0b2-5a96-47ee-ac90-ff2764d9cc65","manifestID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9","bundledeployment":{"metadata":{"name":"fleet-agent-local","namespace":"cluster-fleet-local-local-1a3d67d0a899","creationTimestamp":null,"labels":{"<http://fleet.cattle.io/bundle-name|fleet.cattle.io/bundle-name>":"fleet-agent-local","<http://fleet.cattle.io/bundle-namespace|fleet.cattle.io/bundle-namespace>":"fleet-local","<http://fleet.cattle.io/cluster|fleet.cattle.io/cluster>":"local","<http://fleet.cattle.io/cluster-namespace|fleet.cattle.io/cluster-namespace>":"fleet-local","<http://fleet.cattle.io/managed|fleet.cattle.io/managed>":"true"},"finalizers":["<http://fleet.cattle.io/bundle-deployment-finalizer|fleet.cattle.io/bundle-deployment-finalizer>"]},"spec":{"stagedOptions":{"defaultNamespace":"cattle-fleet-local-system","helm":{"takeOwnership":true}},"stagedDeploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","options":{"defaultNamespace":"cattle-fleet-local-system","helm":{"takeOwnership":true}},"deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623"},"status":{"display":{},"resourceCounts":{"ready":0,"desiredReady":0,"waitApplied":0,"modified":0,"orphaned":0,"missing":0,"unknown":0,"notReady":0}}},"deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","operation":"unchanged"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"bundle","msg":"Unchanged bundledeployment","controller":"bundle","controllerGroup":"<http://fleet.cattle.io|fleet.cattle.io>","controllerKind":"Bundle","Bundle":{"name":"fleet-agent-local","namespace":"fleet-local"},"namespace":"fleet-local","name":"fleet-agent-local","reconcileID":"456fda24-a67a-4ec9-b25f-298ea154761c","manifestID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9","bundledeployment":{"metadata":{"name":"fleet-agent-local","namespace":"cluster-fleet-local-local-1a3d67d0a899","creationTimestamp":null,"labels":{"<http://fleet.cattle.io/bundle-name|fleet.cattle.io/bundle-name>":"fleet-agent-local","<http://fleet.cattle.io/bundle-namespace|fleet.cattle.io/bundle-namespace>":"fleet-local","<http://fleet.cattle.io/cluster|fleet.cattle.io/cluster>":"local","<http://fleet.cattle.io/cluster-namespace|fleet.cattle.io/cluster-namespace>":"fleet-local","<http://fleet.cattle.io/managed|fleet.cattle.io/managed>":"true"},"finalizers":["<http://fleet.cattle.io/bundle-deployment-finalizer|fleet.cattle.io/bundle-deployment-finalizer>"]},"spec":{"stagedOptions":{"defaultNamespace":"cattle-fleet-local-system","helm":{"takeOwnership":true}},"stagedDeploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","options":{"defaultNamespace":"cattle-fleet-local-system","helm":{"takeOwnership":true}},"deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623"},"status":{"display":{},"resourceCounts":{"ready":0,"desiredReady":0,"waitApplied":0,"modified":0,"orphaned":0,"missing":0,"unknown":0,"notReady":0}}},"deploymentID":"s-2f332c47bb36e1bc8d70932ee0158e1b3289ae7ef2ea995e2bd77828ef2e9:8a42b4463e55a59ce2ccdf3c53c32455ce5fd0f601587bf57b5624b3cf8bb623","operation":"unchanged"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
{"level":"info","ts":"2025-11-13T20:53:53Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}
(I'll give Rancher a update from 2.12.1 to 2.12 to see if that has any effect).
c
and you haven’t been able to identify what is changing the fleet cluster resource, or what those changes are?
k
No 😞
c
do you have audit logs?
k
No, but I’ll guess that would be my next option after the update
(Will do that tomorrow).
For now many thanks already for your help and support! Really appreciate your help and time
c
the logs you’re seeing all come from this function and all just indicate that the cluster resource keeps changing… have you looked at any of the other fleet pod or container logs?
k
I’ve looked in both the fleet-controller and the fleet-agent. Basically that are the only pods I can identify which are part of the fleet setup. The helmOps pod isn’t doing anything.
We’ve been seeing this issue also only on the ‘local’ cluster. All downstream clusters look healthy and doesn’t seem to be updated
c
you looked at logs from all three containers in the fleet-controller pod? I only saw logs from fleet-agentmanagement. there are two more containers…
k
I've looked at all containers. But to be compleet I'll post them again as separate files. I've added three logs from the different containers in the fleet-controller pod and the logs from a fleet-agent
fleet-controller.txtfleet-cleanup.txtfleet-agent.txtfleet-agentmanagement.txt
We've also reverted our change on the private root CA and upgraded Rancher from 2.12.1 to 2.12.3 without any luck.
249 Views
Open in Slack
PreviousNext
Powered by