Hello all Does anyone know why this `if` is here <https gith rancher-users #general

Title

many-evening-49066

02/02/2023, 2:40 PM

Hello all! Does anyone know why this

if

is here: https://github.com/rancher/charts/blob/dev-v2.7/charts/rancher-monitoring/102.0.0%2Bup40.1.2/templates/alertmanager/secret.yaml#L4 It is not in the official prometheus-operator chart and it makes changes to alertmanager not apply since the secret already exists. I tried git blaming without success

clean-spoon-54156

02/02/2023, 4:36 PM

It’s by design. Noted in the docs in https://ranchermanager.docs.rancher.com/v2.6/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager#multiple-alertmanager-replicas

By default, the Alertmanager Config Secret created by Rancher will never be modified or deleted on an upgrade or uninstall of the rancher-monitoring chart. This restriction prevents users from losing or overwriting their alerting configuration when executing operations on the chart.

The lookup call ensures we do not apply the resource on a re-install if the secret exists to overwrite its contents (which would anyways cause for helm installation to fail, since Helm cannot assume ownership of an existing resource)

many-evening-49066

02/02/2023, 4:38 PM

Thanks, I eventually found it. I suggest there should be a way to change this behavior to force overwriting it. I am installing rancher-monitoring with Helm in our k8s cluster and I want to change the receiver of the Watchdog to deadmansnitch, in a GitOps matter

Meaning my source of truth is git, and I indeed want it to be overwritten. I found a way to mitigate this, by: • disabling the secret creation by enabling this flag:

alertmanager:
  alertmanagerSpec:
    useExistingSecret: true

• creating the secret

alertmanager-rancher-monitoring-alertmanager

in `cattle-monitoring-system`'s namespace with

alertmanager.yaml

and

rancher_defaults.tmpl

content

clean-spoon-54156

02/02/2023, 4:42 PM

I was just about to suggest that 😄. If you’re using Fleet, you can probably use overlays to manage the secret via Git outside the Helm chart as well, ref: https://github.com/rancher/fleet-examples/blob/master/single-cluster/helm-kustomize/fleet.yaml

👍 1

many-evening-49066

02/02/2023, 4:42 PM

mimicking what the helm chart does (here https://github.com/rancher/charts/blob/6083dc0adcdfdbb1bce5daf53bb54da9433d7641/charts/rancher-monitoring/102.0.0+up40.1.2/templates/alertmanager/secret.yaml) but in a manually way

we're using fleet, but we want to change it after installation, to update it

clean-spoon-54156

02/02/2023, 4:44 PM

yes, you can use the same values setting you are using and then manage the secret with the given name in an overlay file instead. That way on updating the secret in the overlay file in Git, you will be able to see it change on upgrades too

many-evening-49066

02/02/2023, 4:44 PM

I have also tried using AlertmanagerConfig crds to do this, but it applies namespace matching to the routes/receivers and prefixes it with namespace as well

do you think it is out of the scope of rancher to offer a flag/boolean (e.g.

alwaysOverwriteSecret

) that when set to true it would always generate the Secret from the config applied in helm values?

I can open a PR for this

clean-spoon-54156

02/02/2023, 4:51 PM

Yes I think it would be out of scope. Because the only way to assume ownership of an existing resource not tracked by a Helm installation (or left behind by a previous install) to my knowledge would be to deploy a Job with kubectl capabilities to manually modify the resource every time you upgrade Introducing and maintaining a whole new workload that applies on a specific flag for upgrade that needs to be maintained, when an workaround already exists for managing the secret via Fleet GitOps, would be too heavyweight imo.

If a fix was simpler than that, feel free to open up a PR! But I recommend going the overlay route first

many-evening-49066

02/02/2023, 4:56 PM

imo GitOps way should be a first class citizen as I believe is the way to go in the future, and the workaround is far from friendly. It requires more maintenance and knowledge to keep it up to date with rancher-monitoring sane defaults but adding some modifications to it. But I am not thinking of another simpler solution to this problem right now

➕ 1

👍 1

Very grateful for your help Arvind! 😄 Have a wonderful day

👍 1

clean-spoon-54156

02/02/2023, 5:00 PM

I used to work on Fleet so 💯% agree 😊 https://rancher-users.slack.com/archives/C3ASABBD1/p1675356994318999?thread_ts=1675348831.603629&channel=C3ASABBD1&message_ts=1675356994.318999

Same to you!

😊 1

8 Views

#general Join Slack