Hi ! I'm trying to export logs via rancher logging...
# kubernetes
g
Hi ! I'm trying to export logs via rancher logging through elasticsearch, kibana. But the log shipping stop after a few hours and also the amount of logs is too low. Any ideas ? while logs apearing in fluentd stdout it gives error suddenly like bellow
Copy code
2021-11-01 12:20:53.424337870 +0000 kubernetes.var.log.containers.mongodb-sharded-shard2-data-1_mongodb-ns_mongodb-9750f796afb6f086b77361e2534b258eff1d12645a4b7a1b5c485be00b3c6499.log: {"log":"{\"t\":{\"$date\":\"2021-11-01T12:20:53.424+00:00\"},\"s\":\"I\",  \"c\":\"NETWORK\",  \"id\":22944,   \"ctx\":\"conn1649\",\"msg\":\"Connection ended\",\"attr\":{\"remote\":\"127.0.0.1:42544\",\"connectionId\":1649,\"connectionCount\":28}}\n","stream":"stdout","time":"2021-11-01T12:20:53.42433787Z","t":{"$date":"2021-11-01T12:20:53.424+00:00"},"s":"I","c":"NETWORK","id":22944,"ctx":"conn1649","msg":"Connection ended","attr":{"remote":"127.0.0.1:42544","connectionId":1649,"connectionCount":28},"kubernetes":{"pod_name":"mongodb-sharded-shard2-data-1","namespace_name":"mongodb-ns","pod_id":"e943154d-2b57-44fd-9d79-46cdcb107f75","labels":{"<http://app.kubernetes.io/component|app.kubernetes.io/component>":"shardsvr","<http://app.kubernetes.io/instance|app.kubernetes.io/instance>":"mongodb-sharded","<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>":"Helm","<http://app.kubernetes.io/name|app.kubernetes.io/name>":"mongodb-sharded","controller-revision-hash":"mongodb-sharded-shard2-data-84fb47cdb5","<http://helm.sh/chart|helm.sh/chart>":"mongodb-sharded-3.9.13","shard":"2","<http://statefulset.kubernetes.io/pod-name|statefulset.kubernetes.io/pod-name>":"mongodb-sharded-shard2-data-1"},"annotations":{"<http://cni.projectcalico.org/containerID|cni.projectcalico.org/containerID>":"6514ae432af0236a09a85f8dc078e7c62b4862f146974b14c8e9b0742e9430a6","<http://cni.projectcalico.org/podIP|cni.projectcalico.org/podIP>":"10.42.1.28/32","<http://cni.projectcalico.org/podIPs|cni.projectcalico.org/podIPs>":"10.42.1.28/32","<http://kubernetes.io/limit-ranger|kubernetes.io/limit-ranger>":"LimitRanger plugin set: cpu, memory request for container mongodb; cpu, memory limit for container mongodb"},"host":"85.111.28.103","container_name":"mongodb","docker_id":"9750f796afb6f086b77361e2534b258eff1d12645a4b7a1b5c485be00b3c6499","container_hash":"bitnami/mongodb-sharded@sha256:9aed1a1330fd5702de695f2c6a8e48197521c12277dd7cf1cfd3315f9dbb0036","container_image":"bitnami/mongodb-sharded:4.4.10-debian-10-r13"}}
2021-11-01 12:20:54 +0000 [warn]: #0 send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.containers.csi-provisioner-76894c847b-4trxd_longhorn-system_csi-provisioner-f959070a74afd5fee622df10a41729ff6639692b14b417fd79bd77dc1b971741.log" time=2021-11-01 08:27:46.731722355 +0000