More metrics

[seveas]

Hi folks!

Based on the conversation we just had in the community zoom call, I'd like to propose adding a few more metrics that we are interested in for more insights in the things people throw at our fluent instances.

• Total processing volume of filters filter: add total processing volume metrics #3450
• Records that fail to be parsed by the configured parser
• Timestamp parsing failures (as a separate metric from the above)
• Kafka output failures (they show up as retries and failed retries, but not output.errors)
• http errors in the kubernetes metadata fetcher
• Errors executing lua code
• Configured parsers that don't exist (mostly from kubernetes annotations)

I'd be happy to contribute the code for these, would love input on how valuable these will be for others and whether there are other metrics that could/should be added.

[edsiper]

hi @seveas !

first of all my apologies since your PR landed in the backlog for so long, we will fix that.

I will add some comments to the PR so we can speed up it processing!

note that we are migrating our metrics interface to cmetrics, it's under heavy use now in Fluent Bit.

[seveas]

Thanks! I'll add the cmetrics implementation. For any future metrics, do we need the double implementation as well, or can we get away with just cmetrics as they are new anyway?

[edsiper]

I think we can open the discussion for that now, I would prefer to deprecate or stop it extension with more metrics so we can migrate

[glacieruk]

Are there any plans to add process metrics as part of the node_exporter similar to https://github.com/prometheus/node_exporter/blob/master/collector/processes_linux.go ?

For the process log based metrics you have to specify a specific process to monitor, but it would be good to get metrics for all processes so for example you could see what might be causing high cpu on a box.