https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# logging
a
This message was deleted.
h
I've never worked with Loki but need to. Is installing Promtail needed on Rancher downstream cluster? Or simply installing Logging operator from Rancher Apps sufficient? Then setup ClusterFlows as you mentioned above ?
b
I'm trying to answer that question! So far it certainly seems like there are only a few pieces missing to have exactly that setup. They really should add loki to the rancher-monitoring chart
➕ 1
Is installing Promtail needed on Rancher downstream cluster
I'm able to get pod logs without promtail, using fluentd to ship them.
👍 1
To deploy loki, what I did is render the helm template locally, modify it to my needs, and then use fleet's kustomize support to patch the rancher-monitoring chart
otherwise the official chart will force install the agent, which isn't needed if you have the logging operator installed
h
nice! thanks for the response.
b
np. good luck
h
well - so deploying loki was easy and setting up flow. but it seems to be VERY slow. I am only sending 1 namespace logs and looking like logs will show up 15-20 minutes later
b
yeah, you have to disable output buffer to get anything quickly, and that will still see around 20s delay
h
giving that a try - 🤞
b
Copy code
{host=~".+"} | json | line_format `{{.message}}` | logfmt | level = `error`
poor man's log tail. adjust that last segment to your needs
h
WOW!! that (output buffer) made such a HUGE difference!
b
lol right?!
h
I was certainly not expecting that big of difference
b
btw, I also set this in the 
rancher-logging
chart
Copy code
global:
  dockerRootDirectory: /var/log
the generated fluentbit config will append 
/containers
. I did this because that's where k3s' built-in containerd will put logs. However, I have no idea if this is actually required. even though 
/var/lib/docker/containers
has nothing in it, it seemed to be working somehow before I added that chart value.
👍 1
@hundreds-evening-84071 i was looking through the fluentd docs and found this
Copy code
flush_mode: immediate
this is better than turning off buffering, and seems to be faster.
I take that back. without disabling buffering, it still holds them forever. I didn't realize it was still getting set
👍 1
h
Hey; have you tried to set it like this:
Copy code
flush_mode: interval
flush_interval: 2s
b
I haven't! but was planning on trying that soon too
bringing up a new cluster today. I'll probably try that out later.
h
Also, do you remember - do you have 1 or 2 replicas for 
rancher-logging-root-fluentd
statefulset?
Copy code
# kubectl get statefulset -n cattle-logging-system
NAME                           READY   AGE
rancher-logging-root-fluentd   1/1     51d
b
1 by default
✔️ 1
9 Views
Open in Slack
PreviousNext
Powered by