Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

es_out: support Upstream Servers with configuration overriding #1143

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

mabrarov
Copy link

Implementation of Upstream feature for the Elasticsearch output plugin.

Refer to fluent/fluent-bit#7608 for the changes in Fluent Bit code.

@github-actions
Copy link

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Sep 24, 2023
@mabrarov
Copy link
Author

This pull request is waiting for review of fluent/fluent-bit#7608

@github-actions github-actions bot removed the Stale label Sep 25, 2023
Copy link

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Dec 24, 2023
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 7611f0e to 25c7642 Compare December 24, 2023 09:14
@github-actions github-actions bot removed the Stale label Dec 25, 2023
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 25c7642 to 7138a28 Compare February 24, 2024 11:09
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 7138a28 to c1092dc Compare August 29, 2024 07:21
@mabrarov mabrarov requested review from a team as code owners August 29, 2024 07:21
Copy link
Contributor

@cnorris-cs cnorris-cs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left suggestions, but I've added my technical writing-based approval based on that to keep this moving.

Primarily, I was looking to help the table be more consistent and readable, while implementing some style-based suggestions for consistency.

@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch 2 times, most recently from a78ebf2 to c620db5 Compare August 29, 2024 13:51
@mabrarov
Copy link
Author

Hi @cnorris-cs,

I applied some (!) parts of your suggestions as well as introduced additional changes for consistency with your suggestions. Please take a look at this pull request one more time when you have time. Some of the changes are implemented in dedicated commits (for ease of revert in case of concerns) - please let me know if I need to squash all commits into a single one.

Thank you for your review.

@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from c620db5 to 23f04ef Compare October 13, 2024 15:28
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 23f04ef to d8bad1b Compare October 25, 2024 04:02
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from d8bad1b to d465436 Compare November 10, 2024 19:25
@lockewritesdocs
Copy link
Contributor

@fluent/fluent-bit-maintainers -- this PR needs a technical review.

@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from d465436 to 6546d15 Compare November 29, 2024 11:58
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 6546d15 to 2d9ae77 Compare December 18, 2024 15:50
@mabrarov mabrarov force-pushed the feature/out_es_upstream_support_extended branch from 2d9ae77 to 8448868 Compare March 9, 2025 15:11
| `Current_Time_Index` | Use current time for index generation instead of message record. | `Off` | Yes |
| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or later [doesn't support mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), which requires this value to be `On`. | `Off` | Yes |
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | No |
| `Upstream` | If plugin will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md) documentation section. | _none_ | No |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the only new addition? With the extra column it makes it hard to check

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, mostly new column Overridable was added. Here is the diff without new column:

 | Key | Description | Default |
 | :--- | :--- | :--- |
-| `Host` | IP address or hostname of the target Elasticsearch instance | `127.0.0.1` |
-| `Port` | TCP port of the target Elasticsearch instance | `9200` |
+| `Host` | IP address or hostname of the target Elasticsearch instance. | `127.0.0.1` |
+| `Port` | TCP port of the target Elasticsearch instance. | `9200` |
 | `Path` | Elasticsearch accepts new data on HTTP query path `/_bulk`. You can also serve Elasticsearch behind a reverse proxy on a sub-path. Define the path by adding a path prefix in the indexing HTTP POST URI. | Empty string |
 | `compress` | Set payload compression mechanism. Option available is `gzip`. | _none_ |
 | `Buffer_Size` | Specify the buffer size used to read the response from the Elasticsearch HTTP service. Use for debugging purposes where required to read full responses. Response size grows depending of the number of records inserted. To use an unlimited amount of memory, set this value to `False`. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md). | `512KB` |
 | `Pipeline` | Define which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Fluent Bit side, and avoid pipelines. | _none_ |
 | `AWS_Auth` | Enable AWS Sigv4 Authentication for Amazon OpenSearch Service. | `Off` |
 | `AWS_Region` | Specify the AWS region for Amazon OpenSearch Service. | _none_ |
-| `AWS_STS_Endpoint` | Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service | _none_ |
-| `AWS_Role_ARN` | AWS IAM Role to assume to put records to your Amazon cluster | _none_ |
-| `AWS_External_ID` | External ID for the AWS IAM Role specified with `aws_role_arn` | _none_ |
+| `AWS_STS_Endpoint` | Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service. | _none_ |
+| `AWS_Role_ARN` | AWS IAM Role to assume to put records to your Amazon cluster. | _none_ |
+| `AWS_External_ID` | External ID for the AWS IAM Role specified with `aws_role_arn`. | _none_ |
 | `AWS_Service_Name` | Service name to use in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See [Amazon OpenSearch Serverless](opensearch.md) for more information. | `es` |
-| `AWS_Profile` | AWS profile name | `default` |
-| `Cloud_ID` | If using Elastic's Elasticsearch Service you can specify the `cloud_id` of the cluster running. The string has the format `<deployment_name>:<base64_info>`. Once decoded, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`. | _none_ |
-| `Cloud_Auth` | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud | _none_ |
-| `HTTP_User` | Optional username credential for Elastic X-Pack access | _none_ |
-| `HTTP_Passwd` | Password for user defined in `HTTP_User` | _none_ |
-| `Index` | Index name | `fluent-bit` |
-| `Type` | Type name | `_doc` |
-| `Logstash_Format` | Enable Logstash format compatibility. This option takes a Boolean value: `True/False`, `On/Off` | `Off` |
-| `Logstash_Prefix` | When `Logstash_Format` is enabled, the Index name is composed using a prefix and the date, e.g: If `Logstash_Prefix` is equal to `mydata` your index will become `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | `logstash` |
-| `Logstash_Prefix_Key` | When included: the value of the key in the record will be evaluated as key reference and overrides `Logstash_Prefix` for index generation. If the key/value isn't found in the record then the `Logstash_Prefix` option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md). | _none_ |
-| `Logstash_Prefix_Separator` | Set a separator between `Logstash_Prefix` and date.| `-` |
+| `AWS_Profile` | AWS profile name. | `default` |
+| `Cloud_ID` | If using Elastic's Elasticsearch Service you can specify the `cloud_id` of the cluster running. The string has the format `<deployment_name>:<base64_info>`. After decoding, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`. | _none_ |
+| `Cloud_Auth` | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud. | _none_ |
+| `HTTP_User` | Optional username credential for Elastic X-Pack access. | _none_ |
+| `HTTP_Passwd` | Password for user defined in `HTTP_User`. | _none_ |
+| `Index` | Index name. | `fluent-bit` |
+| `Type` | Type name. | `_doc` |
+| `Logstash_Format` | Enable Logstash format compatibility. This option takes a Boolean value: `True/False`, `On/Off`. | `Off` |
+| `Logstash_Prefix` | When `Logstash_Format` is enabled, the Index name is composed using a prefix and the date, For example, if `Logstash_Prefix` is equal to `mydata`, your index becomes `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | `logstash` |
+| `Logstash_Prefix_Key` | When included: the value of the key in the record will be evaluated as key reference and overrides `Logstash_Prefix` for index generation. If the key/value isn't found in the record, the `Logstash_Prefix` option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md). | _none_ |
+| `Logstash_Prefix_Separator` | Set a separator between `Logstash_Prefix` and date. | `-` |
 | `Logstash_DateFormat` | Time format based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html) to generate the second part of the Index name. | `%Y.%m.%d` |
-| `Time_Key` | When `Logstash_Format` is enabled, each record will get a new timestamp field. The `Time_Key` property defines the name of that field. | `@timestamp` |
+| `Time_Key` | When `Logstash_Format` is enabled, each record gets a new timestamp field. The `Time_Key` property defines the name of that field. | `@timestamp` |
 | `Time_Key_Format` | When `Logstash_Format` is enabled, this property defines the format of the timestamp. | `%Y-%m-%dT%H:%M:%S` |
 | `Time_Key_Nanos` | When `Logstash_Format` is enabled, enabling this property sends nanosecond precision timestamps. | `Off` |
 | `Include_Tag_Key` | When enabled, it append the Tag name to the record. | `Off` |
 | `Tag_Key` | When `Include_Tag_Key` is enabled, this property defines the key name for the tag. | `_flb-key` |
 | `Generate_ID` | When enabled, generate `_id` for outgoing records. This prevents duplicate records when retrying ES. | `Off` |
-| `Id_Key` | If set, `_id` will be the value of the key from incoming record and `Generate_ID` option is ignored. | _none_ |
+| `Id_Key` | If set, `_id` is the value of the key from incoming record, and `Generate_ID` option is ignored. | _none_ |
 | `Write_Operation` | `Write_operation` can be any of: `create`, `index`, `update`, `upsert`. | `create` |
 | `Replace_Dots` | When enabled, replace field name dots with underscore. Required by Elasticsearch 2.0-2.3. | `Off` |
 | `Trace_Output` | Print all ElasticSearch API request payloads to `stdout` for diagnostics. | `Off` |
 | `Trace_Error` | If ElasticSearch returns an error, print the ElasticSearch API request and response for diagnostics. | `Off` |
 | `Current_Time_Index` | Use current time for index generation instead of message record. | `Off` |
-| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or higher [no longer supports mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), and is set to `On`. | `Off` |
+| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or later [doesn't support mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), which requires this value to be `On`. | `Off` |
 | `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |
+| `Upstream` | If plugin will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md) documentation section. | _none_ |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants