@miniature-notebook-6405 Ok from curl statements (--insecure) the Splunk side of the equation seems to be working. How do you debug this stuff? Are there are no fissures to fish between Banzai Flows or Outputs, I have no idea where things are being held up? There are no logs going to Splunk, that's all I know. It might as well be the monolith from Space 1999. It could be the Flow is wrong, it could be the Output is wrong, it could be some container, I have no idea which, can't itself run the http to the Splunk endpoint. You didn't used to have to read the manual to use the logging... and now we need logs to debug the logging.

Hi anyone, I have a question regarding th flow filters - I try to change the output with record_modifier, it works for some, but I haev problems with the syntax for nested records: I want to change the container.name like this:

- record_modifier:

records:

- container.name: ${record["kubernetes"]["container_name"]}

But as Output I just get NULL, so the expression is obviously wrong - but I couldn't figure out how to set it to the correct value...

Hi. Any ideas why I cannot use syslog_rfc5424 formater type like here defined https://github.com/acquia/fluent-plugin-syslog_rfc5424 Here is CRD https://github.com/rancher/charts/blob/dev-v2.6/charts/rancher-logging-crd/100.1.3[…]2Bup3.17.7/templates/logging.banzaicloud.io_clusteroutputs.yaml and no syslog available.

Ok I am using 100.1.2%2Bup3.17.4 but no big difference

I'm curious how others deal with this. With rancher-logging it seems that if one logging flow gets messed up then logging is messed up for all flows. How do you debug to tell which logging flow is the faulty one?

Here is an example of my fluentd logs but there is no way to tell which flow is at fault

Hi! I have problem with rancher logging system. I use loki as a logging service + rancher logging by banzai cloud. My logs are up to 60mins behind the newest one and i can't figure out why. Have someone other similiar problem?

I'm testing out the Banzai logging operator. I have it successfully sending pod logs to my ELK stack. However, is there any way it can be configured to send kubelet container logs to ELK also? Keep in mind that 'kubelet' (and others, like etcd, kube-proxy) are NOT Kubernetes pods, but are rke-managed containers.

hi folks, here with a similar problem to @billions-lifeguard-63461 above. I set up a ClusterOutput going to a fluentd I maintain inside the cluster, on a debugging port (8001 -> forward input plugin that tags it with a prefix debug-stdout, and match debug-stdout goes straight to a @type stdout output). I see containers generating logs steadily but my fluentd isn't seeing anything from the Banzai Cloud fluentd/fluent-bit instances for.... 5-10+ minutes at a time

Almost wondering if the fluent-bit Inotify piece is broken. Are there prometheus metrics produced by the components of the Banzai Cloud system I should look for?

(Note that sometimes, killing the fluent-bit pods and letting them restart suddenly results in a flood of backlogged messages hitting my fluentd, so I suspect something's up with the notification of log changes)

(also to mention- RKE1 cluster running on RHEL 7.9)

FYI- I figured out my problem above. Turns out it was not fluent-bit at all, but fluentd talking via "forward" protocol to my own fluentd where our routing rules reside.

The default buffering with banzaicloud's logging fluentd is to use "lazy" buffer flushing, with a timekey of 10 minutes, so it would "flush" every 10 minutes or so.

This can be adjusted in your Output or ClusterOutput configuration:

apiVersion: <http://logging.banzaicloud.io/v1beta1|logging.banzaicloud.io/v1beta1>

kind: ClusterOutput

metadata:

name: fluentd-logging

namespace: cattle-logging-system

spec:

forward:

servers:

- host: fluentd.prh-logging.svc.cluster.local

port: 8000

heartbeat_type: none

transport: tcp

buffer:

flush_mode: interval

flush_interval: 5s

flush_thread_count: 4

queued_chunks_limit_size: 300

In our case we're still using fluentd v1.13, which does not support Heartbeat, and by default banzaicloud's fluentd (v1.14) has heartbeat enabled over UDP for the port, so we had to disable that. No TLS in our instance either, but the

buffer:

section is the important piece. This enforces flushing every 5 seconds for near-realtime routing of the logs.

Hi, I'm trying to view logs for a specific pod but I am unable to access them via kubeclt or the rancher desktop ui.

Hello All : I am having issues with fluentbit failing to rediscover fluentd service after fluentd restart. I am using banzai logging operator with Rancher 2.7. I have reviewed similar issues , updated fluentbit configuration with suggestions to for SVC DNS query. Appreciate help figuring out what is the issue. FluentBit relevant config

fluentbit:
   dnsConfig:
     options:
     - name: ndots
       value: "2"
   image:
     pullPolicy: Always
     repository: fluent-bit
   inputTail:
     Buffer_Chunk_Size: 1MB
     Buffer_Max_Size: 5MB
     storage.type: filesystem
   network:
     dnsMode: TCP
     dnsPreferIpv4: true
     keepaliveMaxRecycle: 200

Fluentbit strace while fluent bit is failing to discover fluentd service. https://gist.github.com/kingnarmer/b7cef66798a38e833798522fe17eeea8 List of similar issues on github https://github.com/fluent/fluent-bit/issues/5312 https://github.com/fluent/fluent-bit/issues/3581 https://github.com/fluent/fluent-bit/issues/3308 https://github.com/fluent/fluent-bit/issues/2274 https://github.com/fluent/fluent-bit/issues/6010

I am unable to view the logs of my pods. I used to be able to, and I have checked that my user-account should have privileges to view logs (if there are special privileges for that).

@eager-vr-25932 That worked, thank you! It also works if you just NULL out the value field in logs-range instead of deleting the whole logs-range object.

Hi, I am currently trying out the logging in Rancher 2.7. Somehow I can't get the log events to arrive at Loki. I don't see any error messages in the logs of fluentd and fluent. Surely I am making basic mistakes. Does anyone have a tip for me? The kubernetes deployment is here: https://github.com/torsten-liermann/RancherLoggingAndMetrics I have no idea how to debug the logging. Thanks!

I see logging operator not being updated for a while now. Do you have alternate plans for logging going forward ? I am running into many issues with current one.

Hi all, I have a question about custom configuration of the rancher-logging chart. For example, I like to configure how and where fluentd/fluent-bit collects log events or add a syslog output plugin to fluent-bit. I do not understand right now, how to achieve these custom configuration with the rancher charts? Does anybody have a suggestion how to customize fluentd/fluent-bit configs?

maybe just rolling vanilla fluent would be better? 🤔

even chatgpt be like ¯\_(ツ)_/¯

my guess here is rancher uses banzaicloud which has a ton of limitations compared to the standard fluentd operators and folks trying to use standard fluent api/crds cant which causes all kinds of pain

my recommendation would be to ditch the built in rancher logging and roll your own fluentbit/d operators and build things up using those properly formed crds

If I’m trying to get logs from the kube-apiserver, etcd, kubelet, kube-controller-manager containers on the control plane nodes, is there a prescribed method for doing so? I’ve tried following this https://www.suse.com/support/kb/doc/?id=000020959 but maybe I’m missing a step there, but it doesn’t seem to actually work. That essentially sets up fluent-bit on control-plane node (after some other undocumented work) tailing docker container logs in their symlinked location (which is only symlinked for k8s pods, not the external containers like above.