https://rancher.com/ logo
m

miniature-notebook-6405

05/04/2022, 9:35 PM
@miniature-notebook-6405 Ok from curl statements (--insecure) the Splunk side of the equation seems to be working. How do you debug this stuff? Are there are no fissures to fish between Banzai Flows or Outputs, I have no idea where things are being held up? There are no logs going to Splunk, that's all I know. It might as well be the monolith from Space 1999. It could be the Flow is wrong, it could be the Output is wrong, it could be some container, I have no idea which, can't itself run the http to the Splunk endpoint. You didn't used to have to read the manual to use the logging... and now we need logs to debug the logging.
r

refined-battery-60852

05/05/2022, 10:05 PM
I am in the same boat with Rancher Logging chart. It is really hard to debug and find a problem and running issues where I think everything is configured correctly but unable to find a log or something is wrong with debug enabled. I am trying to send to splunk also. I am at the point to send to something locally to just see if it is working.
m

miniature-notebook-6405

05/09/2022, 1:52 PM
@refined-battery-60852 I did find a log, in the Rancher's fluentd logging component, the other components have no logs. It was looking for a ca even though "insecure" was checked, so I uploaded a ca. I started deleting and recreating some Flows and ClusterFlows, and extra outputs, and it started working some of the time. I think the Flow components might interfere with each other.
r

refined-battery-60852

05/09/2022, 1:54 PM
which CA would that be… Cluster or CA of Splunk endpoint?
m

miniature-notebook-6405

05/09/2022, 1:55 PM
My posts are kinda ranty looking back over them...apparently this is rage-related
CA of splunk endpoint
r

refined-battery-60852

05/09/2022, 1:55 PM
mine too
thanks for the help
👍 1
m

miniature-notebook-6405

05/09/2022, 5:09 PM
@refined-battery-60852 You might also try clicking the Update button in the logging operator chart page. That helped. However you would think that everything in the "Events" panel on the cluster page would show up in Splunk, I don't know if that's a valid expectation. Splunk is sure laggy and missing things.
I'm not sure what the utility of this thing is, the regular Rancher UI already has everything structured, you go in and can drill down into application logs, each component is separated already. With Splunk everything is combined into one single firehose and has to be separated again, and yet developers think this thing is going to wipe their butts for them. (more rage)
5 Views