https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
c
It’s by design. Noted in the docs in https://ranchermanager.docs.rancher.com/v2.6/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager#multiple-alertmanager-replicas
Copy code
By default, the Alertmanager Config Secret created by Rancher will never be modified or deleted on an upgrade or uninstall of the rancher-monitoring chart. This restriction prevents users from losing or overwriting their alerting configuration when executing operations on the chart.
The lookup call ensures we do not apply the resource on a re-install if the secret exists to overwrite its contents (which would anyways cause for helm installation to fail, since Helm cannot assume ownership of an existing resource)
m
Thanks, I eventually found it. I suggest there should be a way to change this behavior to force overwriting it. I am installing rancher-monitoring with Helm in our k8s cluster and I want to change the receiver of the Watchdog to deadmansnitch, in a GitOps matter
Meaning my source of truth is git, and I indeed want it to be overwritten. I found a way to mitigate this, by: • disabling the secret creation by enabling this flag:
Copy code
alertmanager:
  alertmanagerSpec:
    useExistingSecret: true
• creating the secret 
alertmanager-rancher-monitoring-alertmanager
in `cattle-monitoring-system`'s namespace with 
alertmanager.yaml
and 
rancher_defaults.tmpl
content
c
I was just about to suggest that 😄. If you’re using Fleet, you can probably use overlays to manage the secret via Git outside the Helm chart as well, ref: https://github.com/rancher/fleet-examples/blob/master/single-cluster/helm-kustomize/fleet.yaml
👍 1
m
we're using fleet, but we want to change it after installation, to update it
c
yes, you can use the same values setting you are using and then manage the secret with the given name in an overlay file instead. That way on updating the secret in the overlay file in Git, you will be able to see it change on upgrades too
m
I have also tried using AlertmanagerConfig crds to do this, but it applies namespace matching to the routes/receivers and prefixes it with namespace as well
do you think it is out of the scope of rancher to offer a flag/boolean (e.g. 
alwaysOverwriteSecret
) that when set to true it would always generate the Secret from the config applied in helm values?
I can open a PR for this
c
Yes I think it would be out of scope. Because the only way to assume ownership of an existing resource not tracked by a Helm installation (or left behind by a previous install) to my knowledge would be to deploy a Job with kubectl capabilities to manually modify the resource every time you upgrade Introducing and maintaining a whole new workload that applies on a specific flag for upgrade that needs to be maintained, when an workaround already exists for managing the secret via Fleet GitOps, would be too heavyweight imo.
If a fix was simpler than that, feel free to open up a PR! But I recommend going the overlay route first
m
imo GitOps way should be a first class citizen as I believe is the way to go in the future, and the workaround is far from friendly. It requires more maintenance and knowledge to keep it up to date with rancher-monitoring sane defaults but adding some modifications to it. But I am not thinking of another simpler solution to this problem right now
➕ 1
👍 1
Very grateful for your help Arvind! 😄 Have a wonderful day
👍 1
c
Same to you!
😊 1
12 Views
Open in Slack
PreviousNext
Powered by