https://rancher.com/ logo
Title
p

polite-nail-80344

03/20/2023, 1:39 PM
If I’m trying to get logs from the kube-apiserver, etcd, kubelet, kube-controller-manager containers on the control plane nodes, is there a prescribed method for doing so? I’ve tried following this https://www.suse.com/support/kb/doc/?id=000020959 but maybe I’m missing a step there, but it doesn’t seem to actually work. That essentially sets up fluent-bit on control-plane node (after some other undocumented work) tailing docker container logs in their symlinked location (which is only symlinked for k8s pods, not the external containers like above.
p

polite-piano-74233

03/20/2023, 5:22 PM
that was the exact doc i was going to recommend, fwiw that setup works for me and i get those logs now (they are very VERY spammy btw)
did you verify your clusteroutput is working correctly?
also make sure your fluentbit daemonset is running on the control nodes
p

polite-nail-80344

03/20/2023, 5:23 PM
it is running on the control nodes, and unforunately I only have those containers setup to go to the clusteroutput.
from looking at the fluent-bit config that gets generated I’m failing to see how this could possible work. the control-plane components don’t run under kubernetes so the files it’s configured to tail don’t exist and the cluster wouldn’t know about those components to add the metadata
p

polite-piano-74233

03/20/2023, 5:36 PM
fluentbit runs on my control nodes, what do you mean by the control-plane components?
ah you know what, maybe it works for me because i used rke1 not rke2?
p

polite-nail-80344

03/20/2023, 5:38 PM
control-plane components being
etcd
kube-apiserver
kubelet
etc
p

polite-piano-74233

03/20/2023, 5:43 PM
they are running on the actual control plane nodes if you do a docker ps
so in theory fluentbit could track them based on the containerd socket or w/e
thats for rke1 though, rke2 i think uses its own special container engine thingy
p

polite-nail-80344

03/20/2023, 5:46 PM
but the generated config is this
[SERVICE]
    Flush        1
    Grace        5
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf
    Coro_Stack_Size    24576
    storage.path  /buffers

[INPUT]
    Name         tail
    DB  /tail-db/tail-containers-state.db
    DB.locking  true
    Mem_Buf_Limit  5MB
    Parser  docker
    Path  /var/log/containers/*.log
    Refresh_Interval  5
    Skip_Long_Lines  On
    Tag  kubernetes.*
[FILTER]
    Name        kubernetes
    Buffer_Size  0
    Kube_CA_File  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Tag_Prefix  kubernetes.var.log.containers
    Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Token_TTL  600
    Kube_URL  <https://kubernetes.default.svc:443>
    Match  kubernetes.*
    Merge_Log  On
    Use_Kubelet  Off
[OUTPUT]
    Name          forward
    Match         *
    Host          myfluentd.logging-operator.svc.cluster.local
    Port          24240

    Retry_Limit  False
so it’s only ever tailing logs under /var/log/containers and then trying to use the kubernetes api to determine their metadata
the files under /var/log/containers/* first only exist for containers run under kubernetes (they are symlinks) and second are just random (not really random) strings with no included metadata, you’d have to query docker via a lua script to (per all examples I’ve found) to find out the names
are you using json-file or journald for your log driver?
im on rke1 btw