(also to mention- RKE1 cluster running on RHEL 7.9)

FYI- I figured out my problem above. Turns out it was not fluent-bit at all, but fluentd talking via "forward" protocol to my own fluentd where our routing rules reside.

The default buffering with banzaicloud's logging fluentd is to use "lazy" buffer flushing, with a timekey of 10 minutes, so it would "flush" every 10 minutes or so.

This can be adjusted in your Output or ClusterOutput configuration:

apiVersion: <http://logging.banzaicloud.io/v1beta1|logging.banzaicloud.io/v1beta1>

kind: ClusterOutput

metadata:

name: fluentd-logging

namespace: cattle-logging-system

spec:

forward:

servers:

- host: fluentd.prh-logging.svc.cluster.local

port: 8000

heartbeat_type: none

transport: tcp

buffer:

flush_mode: interval

flush_interval: 5s

flush_thread_count: 4

queued_chunks_limit_size: 300

In our case we're still using fluentd v1.13, which does not support Heartbeat, and by default banzaicloud's fluentd (v1.14) has heartbeat enabled over UDP for the port, so we had to disable that. No TLS in our instance either, but the

buffer:

section is the important piece. This enforces flushing every 5 seconds for near-realtime routing of the logs.

Hi, I'm trying to view logs for a specific pod but I am unable to access them via kubeclt or the rancher desktop ui.

Hello All : I am having issues with fluentbit failing to rediscover fluentd service after fluentd restart. I am using banzai logging operator with Rancher 2.7. I have reviewed similar issues , updated fluentbit configuration with suggestions to for SVC DNS query. Appreciate help figuring out what is the issue. FluentBit relevant config

fluentbit:
   dnsConfig:
     options:
     - name: ndots
       value: "2"
   image:
     pullPolicy: Always
     repository: fluent-bit
   inputTail:
     Buffer_Chunk_Size: 1MB
     Buffer_Max_Size: 5MB
     storage.type: filesystem
   network:
     dnsMode: TCP
     dnsPreferIpv4: true
     keepaliveMaxRecycle: 200

Fluentbit strace while fluent bit is failing to discover fluentd service. https://gist.github.com/kingnarmer/b7cef66798a38e833798522fe17eeea8 List of similar issues on github https://github.com/fluent/fluent-bit/issues/5312 https://github.com/fluent/fluent-bit/issues/3581 https://github.com/fluent/fluent-bit/issues/3308 https://github.com/fluent/fluent-bit/issues/2274 https://github.com/fluent/fluent-bit/issues/6010

I am unable to view the logs of my pods. I used to be able to, and I have checked that my user-account should have privileges to view logs (if there are special privileges for that).

@eager-vr-25932 That worked, thank you! It also works if you just NULL out the value field in logs-range instead of deleting the whole logs-range object.

Hi, I am currently trying out the logging in Rancher 2.7. Somehow I can't get the log events to arrive at Loki. I don't see any error messages in the logs of fluentd and fluent. Surely I am making basic mistakes. Does anyone have a tip for me? The kubernetes deployment is here: https://github.com/torsten-liermann/RancherLoggingAndMetrics I have no idea how to debug the logging. Thanks!

I see logging operator not being updated for a while now. Do you have alternate plans for logging going forward ? I am running into many issues with current one.

Hi all, I have a question about custom configuration of the rancher-logging chart. For example, I like to configure how and where fluentd/fluent-bit collects log events or add a syslog output plugin to fluent-bit. I do not understand right now, how to achieve these custom configuration with the rancher charts? Does anybody have a suggestion how to customize fluentd/fluent-bit configs?

maybe just rolling vanilla fluent would be better? 🤔

even chatgpt be like ¯\_(ツ)_/¯

my guess here is rancher uses banzaicloud which has a ton of limitations compared to the standard fluentd operators and folks trying to use standard fluent api/crds cant which causes all kinds of pain

my recommendation would be to ditch the built in rancher logging and roll your own fluentbit/d operators and build things up using those properly formed crds

If I’m trying to get logs from the kube-apiserver, etcd, kubelet, kube-controller-manager containers on the control plane nodes, is there a prescribed method for doing so? I’ve tried following this https://www.suse.com/support/kb/doc/?id=000020959 but maybe I’m missing a step there, but it doesn’t seem to actually work. That essentially sets up fluent-bit on control-plane node (after some other undocumented work) tailing docker container logs in their symlinked location (which is only symlinked for k8s pods, not the external containers like above.

Hi, I have a scenario, I have N1 to N10 node and 25 microservices are running, so i need to logg ecah 25 microservices log which going to deploy in different node…. the question is, suppose MicroService-1 (MS1) deployed to N1 and the pod restarted and deployed to N10) (conti)

am i just really new to all this or is logging in general a really difficult syntax to get right

I finally got logging to do a thing! https://blog.k3s.live/rancher-kubernetes-log-forwarding/

(pull kubernetes metadata up to the top log level for ingestion with splunk)

I've decided to move from using flow/output configuration on a per application basis to clusterflow/clusteroutput configuration on a per cluster basis. The second being a more infrastructure oriented approach. What are your preferences and why?

Hey folks! 👋 I'm one of the creators and core maintainers of the Logging Operator. The operator is now hosted under the kube-logging organisation, guided by Axoflow (founded by former members of Banzaicloud and Balabit) and Cisco (owner of the Banzaicloud legacy) and is getting a new momentum. I would be happy to chat with someone to tell a little bit about the future plans around the project and to understand more about Rancher's plans around logging. I would like to make sure the direction and decisions we make are synchronised. Please hit me up here or over our Discord!

Hello everyone 🙂 I installed the logging App via the Marketplace, but we get the following error: Error: container has runAsNonRoot and image has non-numeric user (fluent), cannot verify user is non-root (pod: "rancher-logging-root-fluentd-0_cattle-logging-system(c548a56f-6fbf-4f2e-be3c-5fe2f25b5dea)", container: fluentd) Thanks and regards Fabio

Hi all, when using the logging-operator from the rancher-logging chart will the logs end up double in Loki? (through the ClusterFlow and my custom Flow)? or is this handled somehow?

I am not having any logs in the fluentd pod it that normal ?

Hi Everyone! I have some questions regarding the rancher monitoring helm chart. I've set this up with the thanos sidecar running and grafana as an interface. Is it possible to set up the thanos querier/store within the same helm chart? If not, what would be the best practice of setting up the thanos querier to work with the rancher monitoring setup.

@eager-hair-74809 has left the channel

Hey, would the maintainers be open to collaborate on upgrading the logging operator in Rancher? If so please reach out!