This message was deleted.

That indicates that the alerting plugin has panicked. In the gateway pod there should be some logs that describe what the panic was, if you can find those it would be really helpful to figure out what broke. To fix the issue the gateway pod needs to be restarted; you can delete it and let it get recreated which should solve the problem.

After such a fatal panic, should the pod exit and allow k8s to redeploy it?

No, it's an intentional design that problems in the plugins won't impact the wider running system.

are these the relevant logs https://github.com/rancher/opni/issues/1719 ? These look like the alertmanager logs, but if the code in the alerting gateway is also crashing it might be helpful to have logs from there as well

@bright-oil-36284 Greetings! I updated the issue with additional reproduction information. Unfortunately, I no longer have access to the old gateway logs that would accompany these alertmanager logs. If you can answer the question I posed in the issue re: re-installing Alerting, I can try and reproduce again. It seems this issue makes Alerting unusable for us at the moment.

I'll follow up with further updates on my progress on the github issue

👍