limited-potato-16824
10/12/2022, 3:00 PMhelm list
that REVISION
is increased a couple of times and then the helm chart for fleet-agent is removed and it all repeats again. The cycle time of installing fleet-agent with helm and then bumping REVISION
and uninstalling the helm chart for fleet-agent is ~10 seconds.
It should be said that we have used this cluster for upgrade-tests earlier that might have caused this to happen, but if I can get some hints where I might find the cause for this I will be very grateful since I have spent too much time going round in circles.
Note:
1. The fleet-agent pod is not restarted, but the fleet-agent bundle is jumping between states "Pending" and "Wait applied".
2. We are managing 12 clusters with this Rancher installation and this k3s cluster is the only one fighting the fleet-agent bundle.
fleet-agent time="2022-10-12T14:38:44Z" level=error msg="error syncing 'cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd': handler bundle-deploy: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a70272d3-e2fb-4864-aa9f-a7565c7570b4, UID in object meta: , handler bundle-monitor: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a70272d3-e2fb-4864-aa9f-a7565c7570b4, UID in object meta: , requeuing"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="Helm: Deleting release fleet-agent-k3s-rnd 1"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="Helm: Installing fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:46Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="Helm: Deleting release fleet-agent-k3s-rnd 1"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="Helm: Installing fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=info msg="getting history for release fleet-agent-k3s-rnd"
fleet-agent time="2022-10-12T14:38:48Z" level=error msg="error syncing 'cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd': handler bundle-deploy: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aae9c7b4-452f-418a-a5bc-4a43b6f2fa66, UID in object meta: , handler bundle-monitor: Operation cannot be fulfilled on <http://bundledeployments.fleet.cattle.io|bundledeployments.fleet.cattle.io> \"fleet-agent-k3s-rnd\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundledeployments/cluster-fleet-default-k3s-rnd-11e91fbf4d78/fleet-agent-k3s-rnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: aae9c7b4-452f-418a-a5bc-4a43b6f2fa66, UID in object meta: , requeuing"
quick-sandwich-76600
10/14/2022, 6:46 AMlimited-potato-16824
10/14/2022, 8:52 AMquick-sandwich-76600
10/14/2022, 1:51 PMlimited-potato-16824
10/14/2022, 1:54 PMquick-sandwich-76600
10/14/2022, 1:57 PMlimited-potato-16824
10/14/2022, 2:04 PMquick-sandwich-76600
10/14/2022, 2:05 PMlimited-potato-16824
10/14/2022, 2:06 PMquick-sandwich-76600
10/20/2022, 9:46 AM