@orange-airplane-98016 has left the channel

after opni chart installed, the opni-admin-dashboard is not found. what can i do to check ?

Is there any documentation about configuring opni in a declarative way, once the charts are installed? I figured out how to use the monitoringclusters CRD, but for example the role and rolebinding CRDs seem to have no effect. The goal is to configure everything from terraform. Thanks!

Where are the agents configuration and RBAC roles and roles bindings stored? I seem to loose them all when backing up and restoring the cluster's namespace using velero.

After deployment, the Opni plugin dashboard in OpenSearch shows no anomalous or normal logs. I suspect it is because the log_type of the control plane is 'workload'. How can I fix this issue?

~I am trying to setup opni but it seems the documentation is out of date https://rancher.github.io/opni-monitoring/installation/ Is there any documentation, which is working?~

I have some errors when installing Opni:
opni-cluster-system opni-agent-0 2/3 CrashLoopBackOff 7 (42s ago)
opni-cluster-system: Recent Operation: helm-operation-wxt2j Failed
exit code: 123

Opni = 0.92 + Rancher 2.73 - Does anyone have an Idea

Is there any article introducing anomaly detection algorithms and algorithm performance? From my shallow understanding, this algorithm is more like whether this log has appeared in normal logs. For example, in my normal logs, there is "start to process something", while in abnormal logs there is "failed to process something", and the current anomaly detection algorithm cannot recognize it.

I wonder why we select the ks-test to detect metrics abnomaly? Is there some analysis between the different metrics abnomaly methods

I see that opni supports cluster-level multi-tenancy. Does it provide any support for Rancher project-level multi-tenancy? For example does it replace or provide any integration with Rancher prometheus-federator for project metrics and Rancher banzaicloud project logging?

Also does the roadmap currently include adding Opni Monitoring-like Access Control to Opni Logging? Today, can I manually accomplish something similar by configuring oidc in Opni Opensearch dashboards similar to how it's done here: https://opster.com/guides/opensearch/opensearch-security/how-to-set-up-single-sign-on-using-openid-connect/. Or will one of the controllers in the architecture overwrite this type of manual configuration?

Finally, is it true that the Opni team has plans to take over support for all things related to Rancher observability including the rancher-monitoring, rancher-logging, prometheus-federator, etc. components/helm charts? If so, it would be great to see a published roadmap for opni that provides insight into how the team sees these capabilities coming together over time.

Is the Opni Logging documentation available here: https://github.com/rancher/opni-docs/blob/v0.10/docs/logging/user-management.md still accurate for version 0.10?

Is there a way to add an AlertManager plugin as an additional Alerting Endpoint to Opni? In particular, I would like to integrate the following Mattermost AlertManager Plugin into Opni: https://github.com/cpanato/mattermost-plugin-alertmanager#readme.

Is Alerting currently integrated with just the Monitoring Backend or can it also be used with the Logging Backend and AIOps? For example, when an AIOps control plane anomaly is detected in a cluster, can an Alerting Alarm be triggered?

Opensearch, Data Prepper, and OpenTelemetry all now claim to support metrics in addition to logs and traces as a unified Opensearch-based solution. Elastic seems to have a similar story. If you are familiar with the Opensearch/Data Prepper and/or Elastic support for metrics, how would you compare them to Cortex? Is it reasonable to expect Cortex to scale better than Opensearch/Elasticsearch for metrics data? Also how would you compare the Opni Alerting foundational components to the alerting capabilities built into Opensearch?

Any plans to allow declarative management of the various alerting resources?

I would like to helm install opni-agent to downstream clusters via gitops. In order to do so, where can I find on the admin cluster the "Manual install information" needed for the downstream agent-config Secret?

Greetings. When I tried to install Logging using the 0.11.1 Opni version, I encountered the following error. Any suggestions for troubleshooting the problem?

[403 Forbidden] {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/task/get] and User [name=internalopni, backend_roles=[], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/task/get] and User [name=internalopni, backend_roles=[], requestedTenant=null]"},"status":403}: opensearch request unsuccessful

Thanks.

Now that support for the Webhook Endpoint is landing in the Backend and UI, is there any chance a new release could be cut in the near future that would allow me to exercise this new feature and provide feedback?

Congrats on the 0.11.2-rc1 release. Are you also planning to publish a helm chart for this rc?

Greetings! According to the docs here: https://github.com/rancher/opni-docs/blob/v0.10/docs/installation/opni/backends.md#grafana. an opni user is expected to create their own ingress resource in order to access grafana from a browser. I tried creating my own ingress but it seems the v4 grafana-operator used internally by opni kept deleting the ingress I created 👎 What's the expected behavior here? BTW, any plans to upgrade to the v5 grafana-operator. An upgrade would at least ease github code searching when troubleshooting.

Hello! I have gateway provider noAuth configured as well as ingress

opni-grafana-ingress

. When I attempt to login to grafana, I am presented with

Sign in with oAuth

. When I select the option, I receive the following error message:

{
  "error": "invalid_request",
  "error_description": "The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed. The 'redirect_uri' parameter does not match any of the OAuth 2.0 Client's pre-registered redirect urls."
}

Greetings! I had a couple of Alarms defined and was in the process of defining an additional Alarm when the Alerting Backend completely reset to an uninstalled state. Now when I try to install, I receive the following error:

connection error: desc = "transport: error while dialing: dial unix /tmp/plugin653687210: connect: connection refused"

Is there a technique to reinitialize the Alerting Backend when it reaches this state?

Hi! Similar to the situation mentioned by @creamy-wolf-46823, we have a gateway provider noAuth configured as well as ingress

opni-grafana-ingress.

When attempting to login, we are also presented with the

Sign in with oAuth

, however, clicking on that redirects us to a page that says

select user

with no users to select from. There is no where to type anything and no menu to select from.

Hi all! We are trying to install the agent on another cluster. After running the helm install command (“`helm -n opni-agent install opni-agent opni/opni-agent ….etc`”), we see the agent installed in the opni dashboard. However, it says disconnected

--

Greetings! Are you able to share what's on the roadmap between now and GA?

Hi. My v0.11.2+ opni-collector-agent is reporting the following errors in its log. Is this expected in this release?

$ k -n opni logs opni-collector-agent-rwsqq -c otel-collector | more
2023-09-28T13:14:40.581Z        info    service/telemetry.go:82 Skipping telemetry setup.       {"address": ":8888", "level": "None"}
2023-09-28T13:14:40.582Z        info    kube/client.go:101      k8s filtering   {"kind": "processor", "name": "k8sattributes", "pipeline": "logs", "labelSelector": "", "fieldSelector": ""}
2023-09-28T13:14:40.582Z        info    memorylimiterprocessor@v0.74.0/memorylimiter.go:113     Memory limiter configured       {"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 250, "spike_limit_mib": 50, "check_interval": 1}
2023-09-28T13:14:40.583Z        info    service/service.go:128  Starting otelcol-custom...      {"Version": "0.74.0", "NumCPU": 16}
2023-09-28T13:14:40.583Z        info    extensions/extensions.go:41     Starting extensions...
2023-09-28T13:14:40.583Z        info    adapter/receiver.go:56  Starting stanza receiver        {"kind": "receiver", "name": "journald/rke2", "data_type": "logs"}
2023-09-28T13:14:40.584Z        info    adapter/receiver.go:56  Starting stanza receiver        {"kind": "receiver", "name": "filelog/rke2", "data_type": "logs"}
2023-09-28T13:14:40.584Z        info    adapter/receiver.go:56  Starting stanza receiver        {"kind": "receiver", "name": "filelog/k8s", "data_type": "logs"}
2023-09-28T13:14:40.588Z        info    service/service.go:145  Everything is ready. Begin running and processing data.
2023-09-28T13:14:40.785Z        info    fileconsumer/file.go:196        Started watching file   {"kind": "receiver", "name": "filelog/rke2", "data_type": "logs", "component": "fileconsumer", "path": "/var/lib/rancher/rke2/agent/logs/kubelet.log"}
2023-09-28T13:14:40.785Z        error   helper/transformer.go:110       Failed to process entry {"kind": "receiver", "name": "filelog/rke2", "data_type": "logs", "operator_id": "time-sev", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "drop", "entry": {"observed_timestamp":"2023-09-28T13:14:40.785813939Z","timestamp":"0001-01-01T00:00:00Z","body":"Log file created at: 2023/09/24 01:29:45","attributes":{"log.file.path":"/var/lib/rancher/rke2/agent/logs/kubelet.log"},"severity":0,"scope_name":""}}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/transformer.go:110|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/transformer.go:110>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ParseWith|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ParseWith>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/parser.go:151|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/parser.go:151>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/parser.go:123|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/parser.go:123>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/parser.go:109|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/parser.go:109>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).Process|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).Process>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/parser/regex/regex.go:110|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/parser/regex/regex.go:110>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/writer.go:64|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/helper/writer.go:64>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emit|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emit>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/input/file/file.go:65|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/operator/input/file/file.go:65>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Reader).ReadToEnd|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Reader).ReadToEnd>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/fileconsumer/reader.go:102|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/fileconsumer/reader.go:102>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/fileconsumer/file.go:151|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/fileconsumer/file.go:151>
2023-09-28T13:14:40.796Z        info    fileconsumer/file.go:196        Started watching file   {"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/calico-system_calico-kube-controllers-857487c55f-hl72v_56de42ad-344f-4530-b31e-95114e2076fb/calico-kube-controllers/11.log"}
2023-09-28T13:14:41.021Z        error   exporterhelper/queued_retry.go:317      Dropping data because sending_queue is full. Try increasing queue_size. {"kind": "exporter", "data_type": "logs", "name": "otlp", "dropped_items": 100}
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send>
        <http://go.opentelemetry.io/collector/exporter@v0.74.0/exporterhelper/queued_retry.go:317|go.opentelemetry.io/collector/exporter@v0.74.0/exporterhelper/queued_retry.go:317>
<http://go.opentelemetry.io/collector/exporter/exporterhelper.NewLogsExporter.func2|go.opentelemetry.io/collector/exporter/exporterhelper.NewLogsExporter.func2>
        <http://go.opentelemetry.io/collector/exporter@v0.74.0/exporterhelper/logs.go:115|go.opentelemetry.io/collector/exporter@v0.74.0/exporterhelper/logs.go:115>
<http://go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs|go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs>
        <http://go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36|go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36>
<http://go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1|go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1>
        <http://go.opentelemetry.io/collector@v0.74.0/processor/processorhelper/logs.go:71|go.opentelemetry.io/collector@v0.74.0/processor/processorhelper/logs.go:71>
<http://go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs|go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs>
        <http://go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36|go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36>
<http://go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1|go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1>
        <http://go.opentelemetry.io/collector@v0.74.0/processor/processorhelper/logs.go:71|go.opentelemetry.io/collector@v0.74.0/processor/processorhelper/logs.go:71>
<http://go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs|go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs>
        <http://go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36|go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36>
<http://go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs|go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs>
        <http://go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36|go.opentelemetry.io/collector/consumer@v0.74.0/logs.go:36>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).consumerLoop|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).consumerLoop>
        <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/adapter/receiver.go:135|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.74.0/adapter/receiver.go:135>
2023-09-28T13:14:41.021Z        error   adapter/receiver.go:137 ConsumeLogs() failed    {"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "error": "sending_queue is full"}

Greetings. I have a

kube-node-not-ready

Prometheus Alarm configured with Query

kube_node_status_condition{job="kube-state-metrics",condition="Ready",status="true"} == 0

associated with a 3 node Kubernetes cluster and an alertmanager-webhook-logger Endpoint. I shutdown one of the cluster nodes and verified that "firing" entries were being added to the log for the alertmanager-webhook-logger as expected. I then restarted the downed cluster node and shutdown a different cluster node. In the logger log, I see a "resolved" entry added to the log 5 minutes after the last "firing" as expected. However, I do not see a new "firing" entry for the other cluster node that was shutdown. Any ideas why this might be the case?