Skip to content

Commit 864f352

Browse files
0x006EA1E50x006EA1E5
authored
out_loki: add structured_metadata_map docs (#1527)
Signed-off-by: Greg Eales <[email protected]>
1 parent fbcbc52 commit 864f352

File tree

1 file changed

+75
-24
lines changed

1 file changed

+75
-24
lines changed

‎pipeline/outputs/loki.md

+75-24
Original file line numberDiff line numberDiff line change
@@ -10,28 +10,29 @@ Be aware there is a separate Golang output plugin provided by [Grafana](https://
1010

1111
## Configuration Parameters
1212

13-
| Key | Description | Default |
14-
| :--- | :--- | :--- |
15-
| host | Loki hostname or IP address. Do not include the subpath, i.e. `loki/api/v1/push`, but just the base hostname/URL. | 127.0.0.1 |
16-
| uri | Specify a custom HTTP URI. It must start with forward slash.| /loki/api/v1/push |
17-
| port | Loki TCP port | 3100 |
18-
| tls | Use TLS authentication | off |
19-
| http\_user | Set HTTP basic authentication user name | |
20-
| http\_passwd | Set HTTP basic authentication password | |
21-
| bearer\_token | Set bearer token authentication token value. | |
22-
| header | Add additional arbitrary HTTP header key/value pair. Multiple headers can be set. | |
23-
| tenant\_id | Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent. | |
24-
| labels | Stream labels for API request. It can be multiple comma separated of strings specifying `key=value` pairs. In addition to fixed parameters, it also allows to add custom record keys \(similar to `label_keys` property\). More details in the Labels section. | job=fluent-bit |
25-
| label\_keys | Optional list of record keys that will be placed as stream labels. This configuration property is for records key only. More details in the Labels section. | |
26-
| label\_map\_path | Specify the label map file path. The file defines how to extract labels from each record. More details in the Labels section. | |
27-
| structured\_metadata | Optional comma-separated list of `key=value` strings specifying structured metadata for the log line. Like the `labels` parameter, values can reference record keys using record accessors. See [Structured metadata](#structured-metadata) for more information. | |
28-
| remove\_keys | Optional list of keys to remove. | |
29-
| drop\_single\_key | If set to true and after extracting labels only a single key remains, the log line sent to Loki will be the value of that key in line\_format. If set to `raw` and the log line is a string, the log line will be sent unquoted. | off |
30-
| line\_format | Format to use when flattening the record to a log line. Valid values are `json` or `key_value`. If set to `json`, the log line sent to Loki will be the Fluent Bit record dumped as JSON. If set to `key_value`, the log line will be each item in the record concatenated together \(separated by a single space\) in the format. | json |
31-
| auto\_kubernetes\_labels | If set to true, it will add all Kubernetes labels to the Stream labels | off |
32-
| tenant\_id\_key | Specify the name of the key from the original record that contains the Tenant ID. The value of the key is set as `X-Scope-OrgID` of HTTP header. It is useful to set Tenant ID dynamically. ||
33-
| compress | Set payload compression mechanism. The only available option is gzip. Default = "", which means no compression. ||
34-
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
13+
| Key | Description | Default |
14+
|:--------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------|
15+
| host | Loki hostname or IP address. Do not include the subpath, i.e. `loki/api/v1/push`, but just the base hostname/URL. | 127.0.0.1 |
16+
| uri | Specify a custom HTTP URI. It must start with forward slash. | /loki/api/v1/push |
17+
| port | Loki TCP port | 3100 |
18+
| tls | Use TLS authentication | off |
19+
| http\_user | Set HTTP basic authentication user name | |
20+
| http\_passwd | Set HTTP basic authentication password | |
21+
| bearer\_token | Set bearer token authentication token value. | |
22+
| header | Add additional arbitrary HTTP header key/value pair. Multiple headers can be set. | |
23+
| tenant\_id | Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent. | |
24+
| labels | Stream labels for API request. It can be multiple comma separated of strings specifying `key=value` pairs. In addition to fixed parameters, it also allows to add custom record keys \(similar to `label_keys` property\). More details in the Labels section. | job=fluent-bit |
25+
| label\_keys | Optional list of record keys that will be placed as stream labels. This configuration property is for records key only. More details in the Labels section. | |
26+
| label\_map\_path | Specify the label map file path. The file defines how to extract labels from each record. More details in the Labels section. | |
27+
| structured\_metadata | Optional comma-separated list of `key=value` strings specifying structured metadata for the log line. Like the `labels` parameter, values can reference record keys using record accessors. See [Structured metadata](#structured-metadata) for more information. | |
28+
| structured\_metadata\_map\_keys | Optional comma-separated list of record key strings specifying record values of type map, used to dynamically populate structured metadata for the log line. Values can only reference record keys using record accessors, which should reference map values. Each entry from the referenced map will be used to add an entry to the structured metadata. See [Structured metadata](#structured-metadata) for more information. | |
29+
| remove\_keys | Optional list of keys to remove. | |
30+
| drop\_single\_key | If set to true and after extracting labels only a single key remains, the log line sent to Loki will be the value of that key in line\_format. If set to `raw` and the log line is a string, the log line will be sent unquoted. | off |
31+
| line\_format | Format to use when flattening the record to a log line. Valid values are `json` or `key_value`. If set to `json`, the log line sent to Loki will be the Fluent Bit record dumped as JSON. If set to `key_value`, the log line will be each item in the record concatenated together \(separated by a single space\) in the format. | json |
32+
| auto\_kubernetes\_labels | If set to true, it will add all Kubernetes labels to the Stream labels | off |
33+
| tenant\_id\_key | Specify the name of the key from the original record that contains the Tenant ID. The value of the key is set as `X-Scope-OrgID` of HTTP header. It is useful to set Tenant ID dynamically. | |
34+
| compress | Set payload compression mechanism. The only available option is gzip. Default = "", which means no compression. | |
35+
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
3536

3637
## Labels
3738

@@ -229,7 +230,7 @@ You can get the same behavior this flag provides in Loki with `drop_single_key`
229230
{"job"="fluent-bit"} | json | line_format "{{.log}}"
230231
```
231232
232-
### Structured metadata
233+
### Structured Metadata
233234
234235
[Structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/)
235236
lets you attach custom fields to individual log lines without embedding the
@@ -254,12 +255,62 @@ The following configuration:
254255
structured_metadata pod=$kubernetes['pod_name']
255256
```
256257

257-
258258
Other common uses for structured metadata include trace and span IDs, process and thread IDs, and log levels.
259259

260260
Structured metadata is officially supported starting with Loki 3.0, and shouldn't be used
261261
with Loki deployments prior to Loki 3.0.
262262

263+
#### Structured Metadata Maps
264+
In addition to the `structured_metadata` configuration parameter, a `structured_metadata_map_keys` is available, which can be used to dynamically populate structured metadata from map values in the log record. `structured_metadata_map_keys` can be set with a list of record accessors, where each one should reference map values in the log record. Record accessors which do not match a map value will simply be skipped.
265+
266+
The following configuration is similar to the above, except now all entries in the log record map value `$kubernetes` will be used as structured metadata entries.
267+
268+
{% tabs %}
269+
{% tab title="fluent-bit.conf" %}
270+
```text
271+
[OUTPUT]
272+
name loki
273+
match *
274+
labels cluster=my-k8s-cluster, region=us-east-1
275+
structured_metadata_map_keys $kubernetes
276+
```
277+
{% endtab %}
278+
279+
{% tab title="fluent-bit.yaml" %}
280+
```yaml
281+
outputs:
282+
- name: loki
283+
match: *
284+
labels: cluster=my-k8s-cluster, region=us-east-1
285+
structured_metadata_map_keys: $kubernetes
286+
```
287+
{% endtab %}
288+
{% endtabs %}
289+
290+
Assuming the value `$kubernetes` is a map containing two entries `namespace_name` and `pod_name`, the above configuration is equivalent to:
291+
292+
{% tabs %}
293+
{% tab title="fluent-bit.conf" %}
294+
```text
295+
[OUTPUT]
296+
name loki
297+
match *
298+
labels cluster=my-k8s-cluster, region=us-east-1
299+
structured_metadata $kubernetes['namespace_name'],$kubernetes['pod_name']
300+
```
301+
{% endtab %}
302+
303+
{% tab title="fluent-bit.yaml" %}
304+
```yaml
305+
outputs:
306+
- name: loki
307+
match: *
308+
labels: cluster=my-k8s-cluster, region=us-east-1
309+
structured_metadata: $kubernetes['namespace_name'], $kubernetes['pod_name']
310+
```
311+
{% endtab %}
312+
{% endtabs %}
313+
263314
## Networking and TLS Configuration
264315

265316
This plugin inherit core Fluent Bit features to customize the network behavior and optionally enable TLS in the communication channel. For more details about the specific options available refer to the following articles:

0 commit comments

Comments
 (0)