Consuming data from Elastic beats

[agup006]

Background: Many users who were leveraging Beats log collectors are stuck with Elasticsearch 7.10 and are seeking how they should move forward with the new OpenSearch projects. While OpenSearch has partnered with Calyptia on building new plugins for Fluentd and Fluent Bit, migrating to these technologies takes planning, time and effort.

Fluentd and Fluent Bit have integrated with many ecosystems including logstash, OpenMetrics, and OpenTelemetry, and one proposal is if we can allow for a staged approach of migration by supporting the Lumberjack protocol with Fluentd / Fluent Bit.

We have done this in the past with Fluentd (Splunk HTTP HEC receiver) plugin that allows users who are leveraging Splunk to slowly migrate over to Fluent Bit and Fluentd.

Proposal

Considerations

• What versions / distros of beats should we support?
• Do we want to focus on filebeat / metricbeat or also work with other specifics (E.g. heartbeat)?
• What type of authentication would we need to support on the Fluent Bit / Fluentd side? Token based auth, cert, or other?
• Are there considerations with compression or other data formats that we should be aware of

[agup006]

Additional consideration - how would modules work with this input plugin? What can we add day one and then beyond.