-
Notifications
You must be signed in to change notification settings - Fork 511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
es_out: support Upstream Servers with configuration overriding #1143
base: master
Are you sure you want to change the base?
es_out: support Upstream Servers with configuration overriding #1143
Conversation
cb36f75
to
7611f0e
Compare
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
This pull request is waiting for review of fluent/fluent-bit#7608 |
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
7611f0e
to
25c7642
Compare
25c7642
to
7138a28
Compare
7138a28
to
c1092dc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left suggestions, but I've added my technical writing-based approval based on that to keep this moving.
Primarily, I was looking to help the table be more consistent and readable, while implementing some style-based suggestions for consistency.
a78ebf2
to
c620db5
Compare
Hi @cnorris-cs, I applied some (!) parts of your suggestions as well as introduced additional changes for consistency with your suggestions. Please take a look at this pull request one more time when you have time. Some of the changes are implemented in dedicated commits (for ease of revert in case of concerns) - please let me know if I need to squash all commits into a single one. Thank you for your review. |
c620db5
to
23f04ef
Compare
23f04ef
to
d8bad1b
Compare
d8bad1b
to
d465436
Compare
@fluent/fluent-bit-maintainers -- this PR needs a technical review. |
d465436
to
6546d15
Compare
6546d15
to
2d9ae77
Compare
Signed-off-by: Marat Abrarov <[email protected]>
Signed-off-by: Marat Abrarov <[email protected]>
…d on review comments. Signed-off-by: Marat Abrarov <[email protected]>
2d9ae77
to
8448868
Compare
| `Current_Time_Index` | Use current time for index generation instead of message record. | `Off` | Yes | | ||
| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or later [doesn't support mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), which requires this value to be `On`. | `Off` | Yes | | ||
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | No | | ||
| `Upstream` | If plugin will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md) documentation section. | _none_ | No | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this the only new addition? With the extra column it makes it hard to check
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, mostly new column Overridable
was added. Here is the diff without new column:
| Key | Description | Default |
| :--- | :--- | :--- |
-| `Host` | IP address or hostname of the target Elasticsearch instance | `127.0.0.1` |
-| `Port` | TCP port of the target Elasticsearch instance | `9200` |
+| `Host` | IP address or hostname of the target Elasticsearch instance. | `127.0.0.1` |
+| `Port` | TCP port of the target Elasticsearch instance. | `9200` |
| `Path` | Elasticsearch accepts new data on HTTP query path `/_bulk`. You can also serve Elasticsearch behind a reverse proxy on a sub-path. Define the path by adding a path prefix in the indexing HTTP POST URI. | Empty string |
| `compress` | Set payload compression mechanism. Option available is `gzip`. | _none_ |
| `Buffer_Size` | Specify the buffer size used to read the response from the Elasticsearch HTTP service. Use for debugging purposes where required to read full responses. Response size grows depending of the number of records inserted. To use an unlimited amount of memory, set this value to `False`. Otherwise set the value according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md). | `512KB` |
| `Pipeline` | Define which pipeline the database should use. For performance reasons, it's strongly suggested to do parsing and filtering on Fluent Bit side, and avoid pipelines. | _none_ |
| `AWS_Auth` | Enable AWS Sigv4 Authentication for Amazon OpenSearch Service. | `Off` |
| `AWS_Region` | Specify the AWS region for Amazon OpenSearch Service. | _none_ |
-| `AWS_STS_Endpoint` | Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service | _none_ |
-| `AWS_Role_ARN` | AWS IAM Role to assume to put records to your Amazon cluster | _none_ |
-| `AWS_External_ID` | External ID for the AWS IAM Role specified with `aws_role_arn` | _none_ |
+| `AWS_STS_Endpoint` | Specify the custom STS endpoint to be used with STS API for Amazon OpenSearch Service. | _none_ |
+| `AWS_Role_ARN` | AWS IAM Role to assume to put records to your Amazon cluster. | _none_ |
+| `AWS_External_ID` | External ID for the AWS IAM Role specified with `aws_role_arn`. | _none_ |
| `AWS_Service_Name` | Service name to use in AWS Sigv4 signature. For integration with Amazon OpenSearch Serverless, set to `aoss`. See [Amazon OpenSearch Serverless](opensearch.md) for more information. | `es` |
-| `AWS_Profile` | AWS profile name | `default` |
-| `Cloud_ID` | If using Elastic's Elasticsearch Service you can specify the `cloud_id` of the cluster running. The string has the format `<deployment_name>:<base64_info>`. Once decoded, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`. | _none_ |
-| `Cloud_Auth` | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud | _none_ |
-| `HTTP_User` | Optional username credential for Elastic X-Pack access | _none_ |
-| `HTTP_Passwd` | Password for user defined in `HTTP_User` | _none_ |
-| `Index` | Index name | `fluent-bit` |
-| `Type` | Type name | `_doc` |
-| `Logstash_Format` | Enable Logstash format compatibility. This option takes a Boolean value: `True/False`, `On/Off` | `Off` |
-| `Logstash_Prefix` | When `Logstash_Format` is enabled, the Index name is composed using a prefix and the date, e.g: If `Logstash_Prefix` is equal to `mydata` your index will become `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | `logstash` |
-| `Logstash_Prefix_Key` | When included: the value of the key in the record will be evaluated as key reference and overrides `Logstash_Prefix` for index generation. If the key/value isn't found in the record then the `Logstash_Prefix` option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md). | _none_ |
-| `Logstash_Prefix_Separator` | Set a separator between `Logstash_Prefix` and date.| `-` |
+| `AWS_Profile` | AWS profile name. | `default` |
+| `Cloud_ID` | If using Elastic's Elasticsearch Service you can specify the `cloud_id` of the cluster running. The string has the format `<deployment_name>:<base64_info>`. After decoding, the `base64_info` string has the format `<deployment_region>$<elasticsearch_hostname>$<kibana_hostname>`. | _none_ |
+| `Cloud_Auth` | Specify the credentials to use to connect to Elastic's Elasticsearch Service running on Elastic Cloud. | _none_ |
+| `HTTP_User` | Optional username credential for Elastic X-Pack access. | _none_ |
+| `HTTP_Passwd` | Password for user defined in `HTTP_User`. | _none_ |
+| `Index` | Index name. | `fluent-bit` |
+| `Type` | Type name. | `_doc` |
+| `Logstash_Format` | Enable Logstash format compatibility. This option takes a Boolean value: `True/False`, `On/Off`. | `Off` |
+| `Logstash_Prefix` | When `Logstash_Format` is enabled, the Index name is composed using a prefix and the date, For example, if `Logstash_Prefix` is equal to `mydata`, your index becomes `mydata-YYYY.MM.DD`. The last string appended belongs to the date when the data is being generated. | `logstash` |
+| `Logstash_Prefix_Key` | When included: the value of the key in the record will be evaluated as key reference and overrides `Logstash_Prefix` for index generation. If the key/value isn't found in the record, the `Logstash_Prefix` option will act as a fallback. The parameter is expected to be a [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md). | _none_ |
+| `Logstash_Prefix_Separator` | Set a separator between `Logstash_Prefix` and date. | `-` |
| `Logstash_DateFormat` | Time format based on [strftime](http://man7.org/linux/man-pages/man3/strftime.3.html) to generate the second part of the Index name. | `%Y.%m.%d` |
-| `Time_Key` | When `Logstash_Format` is enabled, each record will get a new timestamp field. The `Time_Key` property defines the name of that field. | `@timestamp` |
+| `Time_Key` | When `Logstash_Format` is enabled, each record gets a new timestamp field. The `Time_Key` property defines the name of that field. | `@timestamp` |
| `Time_Key_Format` | When `Logstash_Format` is enabled, this property defines the format of the timestamp. | `%Y-%m-%dT%H:%M:%S` |
| `Time_Key_Nanos` | When `Logstash_Format` is enabled, enabling this property sends nanosecond precision timestamps. | `Off` |
| `Include_Tag_Key` | When enabled, it append the Tag name to the record. | `Off` |
| `Tag_Key` | When `Include_Tag_Key` is enabled, this property defines the key name for the tag. | `_flb-key` |
| `Generate_ID` | When enabled, generate `_id` for outgoing records. This prevents duplicate records when retrying ES. | `Off` |
-| `Id_Key` | If set, `_id` will be the value of the key from incoming record and `Generate_ID` option is ignored. | _none_ |
+| `Id_Key` | If set, `_id` is the value of the key from incoming record, and `Generate_ID` option is ignored. | _none_ |
| `Write_Operation` | `Write_operation` can be any of: `create`, `index`, `update`, `upsert`. | `create` |
| `Replace_Dots` | When enabled, replace field name dots with underscore. Required by Elasticsearch 2.0-2.3. | `Off` |
| `Trace_Output` | Print all ElasticSearch API request payloads to `stdout` for diagnostics. | `Off` |
| `Trace_Error` | If ElasticSearch returns an error, print the ElasticSearch API request and response for diagnostics. | `Off` |
| `Current_Time_Index` | Use current time for index generation instead of message record. | `Off` |
-| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or higher [no longer supports mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), and is set to `On`. | `Off` |
+| `Suppress_Type_Name` | When enabled, mapping types is removed and `Type` option is ignored. Elasticsearch 8.0.0 or later [doesn't support mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html), which requires this value to be `On`. | `Off` |
| `Workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` |
+| `Upstream` | If plugin will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers](../../administration/configuring-fluent-bit/classic-mode/upstream-servers.md) documentation section. | _none_ |
Signed-off-by: Marat Abrarov <[email protected]>
Implementation of Upstream feature for the Elasticsearch output plugin.
Refer to fluent/fluent-bit#7608 for the changes in Fluent Bit code.