Hi ! I'm trying to export logs via rancher logging...
# kubernetes
g
Hi ! I'm trying to export logs via rancher logging through elasticsearch, kibana. But the log shipping stop after a few hours and also the amount of logs is too low. Any ideas ? while logs apearing in fluentd stdout it gives error suddenly like bellow
Copy code
2021-11-01 12:20:53.424337870 +0000 kubernetes.var.log.containers.mongodb-sharded-shard2-data-1_mongodb-ns_mongodb-9750f796afb6f086b77361e2534b258eff1d12645a4b7a1b5c485be00b3c6499.log: {"log":"{\"t\":{\"$date\":\"2021-11-01T12:20:53.424+00:00\"},\"s\":\"I\",  \"c\":\"NETWORK\",  \"id\":22944,   \"ctx\":\"conn1649\",\"msg\":\"Connection ended\",\"attr\":{\"remote\":\"127.0.0.1:42544\",\"connectionId\":1649,\"connectionCount\":28}}\n","stream":"stdout","time":"2021-11-01T12:20:53.42433787Z","t":{"$date":"2021-11-01T12:20:53.424+00:00"},"s":"I","c":"NETWORK","id":22944,"ctx":"conn1649","msg":"Connection ended","attr":{"remote":"127.0.0.1:42544","connectionId":1649,"connectionCount":28},"kubernetes":{"pod_name":"mongodb-sharded-shard2-data-1","namespace_name":"mongodb-ns","pod_id":"e943154d-2b57-44fd-9d79-46cdcb107f75","labels":{"app.kubernetes.io/component":"shardsvr","app.kubernetes.io/instance":"mongodb-sharded","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"mongodb-sharded","controller-revision-hash":"mongodb-sharded-shard2-data-84fb47cdb5","helm.sh/chart":"mongodb-sharded-3.9.13","shard":"2","statefulset.kubernetes.io/pod-name":"mongodb-sharded-shard2-data-1"},"annotations":{"cni.projectcalico.org/containerID":"6514ae432af0236a09a85f8dc078e7c62b4862f146974b14c8e9b0742e9430a6","cni.projectcalico.org/podIP":"10.42.1.28/32","cni.projectcalico.org/podIPs":"10.42.1.28/32","kubernetes.io/limit-ranger":"LimitRanger plugin set: cpu, memory request for container mongodb; cpu, memory limit for container mongodb"},"host":"85.111.28.103","container_name":"mongodb","docker_id":"9750f796afb6f086b77361e2534b258eff1d12645a4b7a1b5c485be00b3c6499","container_hash":"bitnami/mongodb-sharded@sha256:9aed1a1330fd5702de695f2c6a8e48197521c12277dd7cf1cfd3315f9dbb0036","container_image":"bitnami/mongodb-sharded:4.4.10-debian-10-r13"}}
2021-11-01 12:20:54 +0000 [warn]: #0 send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.containers.csi-provisioner-76894c847b-4trxd_longhorn-system_csi-provisioner-f959070a74afd5fee622df10a41729ff6639692b14b417fd79bd77dc1b971741.log" time=2021-11-01 08:27:46.731722355 +0000