https://rancher.com/ logo
Docs
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
logging
  • m

    miniature-notebook-6405

    05/04/2022, 9:35 PM
    @miniature-notebook-6405 Ok from curl statements (--insecure) the Splunk side of the equation seems to be working. How do you debug this stuff? Are there are no fissures to fish between Banzai Flows or Outputs, I have no idea where things are being held up? There are no logs going to Splunk, that's all I know. It might as well be the monolith from Space 1999. It could be the Flow is wrong, it could be the Output is wrong, it could be some container, I have no idea which, can't itself run the http to the Splunk endpoint. You didn't used to have to read the manual to use the logging... and now we need logs to debug the logging.
    r
    • 2
    • 9
  • g

    gray-camera-14816

    08/03/2022, 9:31 AM
    Hi anyone, I have a question regarding th flow filters - I try to change the output with record_modifier, it works for some, but I haev problems with the syntax for nested records: I want to change the container.name like this:
    - record_modifier:
    records:
    - container.name: ${record["kubernetes"]["container_name"]}
    But as Output I just get NULL, so the expression is obviously wrong - but I couldn't figure out how to set it to the correct value...
  • r

    refined-airline-25231

    08/24/2022, 12:37 PM
    Hi. Any ideas why I cannot use syslog_rfc5424 formater type like here defined https://github.com/acquia/fluent-plugin-syslog_rfc5424 Here is CRD https://github.com/rancher/charts/blob/dev-v2.6/charts/rancher-logging-crd/100.1.3[…]2Bup3.17.7/templates/logging.banzaicloud.io_clusteroutputs.yaml and no syslog available.
  • r

    refined-airline-25231

    08/24/2022, 12:38 PM
    Ok I am using 100.1.2%2Bup3.17.4 but no big difference
  • f

    freezing-holiday-13112

    12/02/2022, 4:18 PM
    I'm curious how others deal with this. With rancher-logging it seems that if one logging flow gets messed up then logging is messed up for all flows. How do you debug to tell which logging flow is the faulty one?
  • f

    freezing-holiday-13112

    12/02/2022, 4:20 PM
    Here is an example of my fluentd logs but there is no way to tell which flow is at fault
  • b

    billions-lifeguard-63461

    01/10/2023, 1:10 PM
    Hi! I have problem with rancher logging system. I use loki as a logging service + rancher logging by banzai cloud. My logs are up to 60mins behind the newest one and i can't figure out why. Have someone other similiar problem?
  • b

    better-vr-95756

    01/10/2023, 10:25 PM
    I'm testing out the Banzai logging operator. I have it successfully sending pod logs to my ELK stack. However, is there any way it can be configured to send kubelet container logs to ELK also? Keep in mind that 'kubelet' (and others, like etcd, kube-proxy) are NOT Kubernetes pods, but are rke-managed containers.
  • w

    wide-actor-79154

    01/19/2023, 4:48 PM
    hi folks, here with a similar problem to @billions-lifeguard-63461 above. I set up a ClusterOutput going to a fluentd I maintain inside the cluster, on a debugging port (8001 -> forward input plugin that tags it with a prefix debug-stdout, and match debug-stdout goes straight to a @type stdout output). I see containers generating logs steadily but my fluentd isn't seeing anything from the Banzai Cloud fluentd/fluent-bit instances for.... 5-10+ minutes at a time
  • w

    wide-actor-79154

    01/19/2023, 4:49 PM
    Almost wondering if the fluent-bit Inotify piece is broken. Are there prometheus metrics produced by the components of the Banzai Cloud system I should look for?
  • w

    wide-actor-79154

    01/19/2023, 4:49 PM
    (Note that sometimes, killing the fluent-bit pods and letting them restart suddenly results in a flood of backlogged messages hitting my fluentd, so I suspect something's up with the notification of log changes)
  • w

    wide-actor-79154

    01/19/2023, 4:50 PM
    (also to mention- RKE1 cluster running on RHEL 7.9)
  • w

    wide-actor-79154

    01/31/2023, 3:21 PM
    FYI- I figured out my problem above. Turns out it was not fluent-bit at all, but fluentd talking via "forward" protocol to my own fluentd where our routing rules reside.
  • w

    wide-actor-79154

    01/31/2023, 3:22 PM
    The default buffering with banzaicloud's logging fluentd is to use "lazy" buffer flushing, with a timekey of 10 minutes, so it would "flush" every 10 minutes or so.
  • w

    wide-actor-79154

    01/31/2023, 3:22 PM
    This can be adjusted in your Output or ClusterOutput configuration:
  • w

    wide-actor-79154

    01/31/2023, 3:22 PM
    apiVersion: <http://logging.banzaicloud.io/v1beta1|logging.banzaicloud.io/v1beta1>
    kind: ClusterOutput
    metadata:
    name: fluentd-logging
    namespace: cattle-logging-system
    spec:
    forward:
    servers:
    - host: fluentd.prh-logging.svc.cluster.local
    port: 8000
    heartbeat_type: none
    transport: tcp
    buffer:
    flush_mode: interval
    flush_interval: 5s
    flush_thread_count: 4
    queued_chunks_limit_size: 300
    🙌 1
    👍 1
    a
    • 2
    • 1
  • w

    wide-actor-79154

    01/31/2023, 3:23 PM
    In our case we're still using fluentd v1.13, which does not support Heartbeat, and by default banzaicloud's fluentd (v1.14) has heartbeat enabled over UDP for the port, so we had to disable that. No TLS in our instance either, but the
    buffer:
    section is the important piece. This enforces flushing every 5 seconds for near-realtime routing of the logs.
  • p

    plain-crayon-98434

    02/03/2023, 4:24 PM
    Hi, I'm trying to view logs for a specific pod but I am unable to access them via kubeclt or the rancher desktop ui.
    • 1
    • 1
  • s

    sparse-ghost-91718

    02/20/2023, 1:27 PM
    Hello All : I am having issues with fluentbit failing to rediscover fluentd service after fluentd restart. I am using banzai logging operator with Rancher 2.7. I have reviewed similar issues , updated fluentbit configuration with suggestions to for SVC DNS query. Appreciate help figuring out what is the issue. FluentBit relevant config
    fluentbit:
       dnsConfig:
         options:
         - name: ndots
           value: "2"
       image:
         pullPolicy: Always
         repository: fluent-bit
       inputTail:
         Buffer_Chunk_Size: 1MB
         Buffer_Max_Size: 5MB
         storage.type: filesystem
       network:
         dnsMode: TCP
         dnsPreferIpv4: true
         keepaliveMaxRecycle: 200
    Fluentbit strace while fluent bit is failing to discover fluentd service. https://gist.github.com/kingnarmer/b7cef66798a38e833798522fe17eeea8 List of similar issues on github https://github.com/fluent/fluent-bit/issues/5312 https://github.com/fluent/fluent-bit/issues/3581 https://github.com/fluent/fluent-bit/issues/3308 https://github.com/fluent/fluent-bit/issues/2274 https://github.com/fluent/fluent-bit/issues/6010
  • s

    stale-hairdresser-10037

    02/21/2023, 4:23 PM
    I am unable to view the logs of my pods. I used to be able to, and I have checked that my user-account should have privileges to view logs (if there are special privileges for that).
    e
    • 2
    • 1
  • s

    stale-hairdresser-10037

    02/22/2023, 4:38 PM
    @eager-vr-25932 That worked, thank you! It also works if you just NULL out the value field in logs-range instead of deleting the whole logs-range object.
    👍 1
  • f

    flat-keyboard-41428

    02/23/2023, 7:27 PM
    Hi, I am currently trying out the logging in Rancher 2.7. Somehow I can't get the log events to arrive at Loki. I don't see any error messages in the logs of fluentd and fluent. Surely I am making basic mistakes. Does anyone have a tip for me? The kubernetes deployment is here: https://github.com/torsten-liermann/RancherLoggingAndMetrics I have no idea how to debug the logging. Thanks!
  • s

    sparse-ghost-91718

    02/28/2023, 12:54 PM
    I see logging operator not being updated for a while now. Do you have alternate plans for logging going forward ? I am running into many issues with current one.
  • c

    chilly-oxygen-20154

    02/28/2023, 9:29 PM
    Hi all, I have a question about custom configuration of the rancher-logging chart. For example, I like to configure how and where fluentd/fluent-bit collects log events or add a syslog output plugin to fluent-bit. I do not understand right now, how to achieve these custom configuration with the rancher charts? Does anybody have a suggestion how to customize fluentd/fluent-bit configs?
    p
    • 2
    • 1
  • p

    polite-piano-74233

    03/15/2023, 6:02 PM
    maybe just rolling vanilla fluent would be better? 🤔
  • p

    polite-piano-74233

    03/15/2023, 6:03 PM
    even chatgpt be like ¯\_(ツ)_/¯
  • p

    polite-piano-74233

    03/17/2023, 1:06 PM
    my guess here is rancher uses banzaicloud which has a ton of limitations compared to the standard fluentd operators and folks trying to use standard fluent api/crds cant which causes all kinds of pain
  • p

    polite-piano-74233

    03/17/2023, 1:06 PM
    my recommendation would be to ditch the built in rancher logging and roll your own fluentbit/d operators and build things up using those properly formed crds
  • p

    polite-nail-80344

    03/20/2023, 1:39 PM
    If I’m trying to get logs from the kube-apiserver, etcd, kubelet, kube-controller-manager containers on the control plane nodes, is there a prescribed method for doing so? I’ve tried following this https://www.suse.com/support/kb/doc/?id=000020959 but maybe I’m missing a step there, but it doesn’t seem to actually work. That essentially sets up fluent-bit on control-plane node (after some other undocumented work) tailing docker container logs in their symlinked location (which is only symlinked for k8s pods, not the external containers like above.
    p
    • 2
    • 16
  • b

    bitter-tailor-6977

    03/20/2023, 3:59 PM
    Hi, I have a scenario, I have N1 to N10 node and 25 microservices are running, so i need to logg ecah 25 microservices log which going to deploy in different node…. the question is, suppose MicroService-1 (MS1) deployed to N1 and the pod restarted and deployed to N10) (conti)
    • 1
    • 3
Powered by Linen
Title
b

bitter-tailor-6977

03/20/2023, 3:59 PM
Hi, I have a scenario, I have N1 to N10 node and 25 microservices are running, so i need to logg ecah 25 microservices log which going to deploy in different node…. the question is, suppose MicroService-1 (MS1) deployed to N1 and the pod restarted and deployed to N10) (conti)
so in above situation, how kube-logging operator Daemonset will be a solution… i need log lifecycle to continued in any dashboard like kibana or loki
so if the pod is recreated in N10 , how i can see the entire logs in kibana or loki dashboard (because now the logs will be taken from N10 Daemonset right)
any solution do we have in rancher logging for this ?
View count: 1