Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Title
b
breezy-action-99129
05/10/2023, 9:52 AMAfter deployment, the Opni plugin dashboard in OpenSearch shows no anomalous or normal logs. I suspect it is because the log_type of the control plane is 'workload'. How can I fix this issue?
The etcd and apiserver log is not collected and how can I fix?
the etcd log is collected normally. But the log type is still workload
f
famous-dusk-36365
05/15/2023, 10:26 PMWhat distribution cluster is this running on?
b
breezy-action-99129
05/17/2023, 7:58 AMon a native k8s. And i think it is because the hasField method in opnipreprocessing plugin is not worked as expected https://github.com/tybalex/opni-ingest-plugin/blob/f1a024e46e1dbca077c5180a3a120f9[…]a/org/opensearch/opnipreprocessing/plugin/OpniPreProcessor.java
f
famous-dusk-36365
05/17/2023, 8:01 AMThat should work if the open telemetry collector is scraping the logs and annotating the logs correctly. Is your cluster dpeloyed by kubeadm? We can do some testing on this
b
breezy-action-99129
05/17/2023, 8:21 AMMaybe you can insert the log to opensearch to find the problem.
POST _bulk
{ "create": { "_index": "logs" } }
{"k8s.container.name":"etcd","k8s.namespace.name":"kube-system","container.image.tag":"60080/tkestack/etcd","log":"{\"level\":\"info\",\"ts\":\"2023-05-11T03:23:28.748Z\",\"caller\":\"mvcc/hash.go:137\",\"msg\":\"storing new hash\",\"hash\":3343241951,\"revision\":5815545,\"compact-revision\":5814139}","k8s.pod.uid":"","Body.value":"2023-05-11T11:23:28.748397999+08:00 stderr F {\"level\":\"info\",\"ts\":\"2023-05-11T03:23:28.748Z\",\"caller\":\"mvcc/hash.go:137\",\"msg\":\"storing new hash\",\"hash\":3343241951,\"revision\":5815545,\"compact-revision\":5814139}","logtag":"F","namespace_name":"kube-system","SeverityNumber":0,"cluster_id":"180a2c2f-63ba-4b8f-a532-ba4a549bcfb1","log_type":"zlc_test","container.image.name":"registry.cn","k8s.pod.name":"etcd-192.168.137.245","stream":"stderr","kubernetes_component":"","k8s.pod.labels.tier":"control-plane","deployment":"","raw_ts":"raw time","k8s.pod.confighash":"29d6e6a1c05a2d07020e80d4b0ad7aae","restart_count":"2","template_matched":"","k8s.pod.labels.component":"etcd","log_source_field":"message","container_image":"registry.alauda.cn","anomaly_level":"","pod_name":"etcd-192.168.137.245","TraceFlags":0,"@timestamp":"2023-05-11T03:23:28.748397999Z","k8s.node.name":"192.168.137.245","ingest_at":"2023-05-11 03:23:44.705","service":"","time":"2023-05-11T11:23:28.748397999+08:00"}
f
famous-dusk-36365
05/17/2023, 9:11 AMI see this is actually happening on RKE2. I'll create an issue to look into this
b
breezy-action-99129
05/18/2023, 7:19 AMI think it is because the dot too and temporarily modified the preprocessor. Thanks for your help
BTW, is any article introduce the log anomaly algorithm?