Skip to content

Conditional processing doc fixes #1600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 13 additions & 9 deletions pipeline/processors/README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,28 @@
# Processors

Processors are components that modify, transform, or enhance data as it flows through Fluent Bit.
Unlike [filters](../filters/README.md), processors are tightly coupled to inputs, which means they
execute immediately and avoid creating a performance bottleneck.
Processors are components that modify, transform, or enhance data as it flows
through Fluent Bit. Unlike [filters](../filters/README.md), processors are
tightly coupled to inputs, which means they execute immediately and avoid
creating a performance bottleneck.

Additionally, filters can be implemented in a way that mimics the behavior of processors, but
processors can't be implemented in a way that mimics filters.
Additionally, filters can be implemented in a way that mimics the behavior of
processors, but processors can't be implemented in a way that mimics filters.

## Available processors

Fluent Bit offers the following processors:

- [Content Modifier](content-modifier.md): Manipulate the content, metadata, and attributes of logs and traces.
- [Content Modifier](content-modifier.md): Manipulate the content, metadata, and
attributes of logs and traces.
- [Labels](labels.md): Add, update, or delete metric labels.
- [Metrics Selector](metrics-selector.md): Choose which metrics to keep or discard.
- [OpenTelemetry Envelope](opentelemetry-envelope.md): Transform logs into an OpenTelemetry-compatible format.
- [OpenTelemetry Envelope](opentelemetry-envelope.md): Transform logs into an
OpenTelemetry-compatible format.
- [SQL](sql.md): Use SQL queries to extract log content.

## Features

All available processors include the following features:
Compatible processors include the following features:

- [Conditional Processing](conditional-processing.md): Apply processors selectively to records that meet specific criteria.
- [Conditional Processing](conditional-processing.md): Selectively apply processors
to logs based on the value of fields that those logs contain.
45 changes: 27 additions & 18 deletions pipeline/processors/conditional-processing.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
# Conditional processing

Conditional processing lets you selectively apply [processors](README.md) to logs based on
the value of fields that those logs contain. This feature lets you create processing pipelines
that only process records that meet certain criteria, and ignore the rest.
Conditional processing lets you selectively apply [processors](README.md) to
logs based on the value of fields that those logs contain. This feature lets you
create processing pipelines that only process records that meet certain
criteria, and ignore the rest.

Conditional processing is available in Fluent Bit version 4.0 and greater.

## Configuration

You can turn a standard processor into a conditional processor by adding a `condition` block to the
processor's YAML configuration settings.
You can turn a standard processor into a conditional processor by adding a
`condition` block to the processor's YAML configuration settings.

{% hint style="info" %}
Conditional processing is only available for
[YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md),
not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).
Conditional processing is only available for [YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md), not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Conditional processing is only available for [YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md), not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).
Conditional processing is only available for [YAML configuration files](../../administration/configuring-fluent-bit/yaml/README.md),
not [classic configuration files](../../administration/configuring-fluent-bit/classic-mode/README.md).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had it like that previously but gitbook got mad so I think it's safer to leave it all on one line :|

image

(no space between "for" and "YAML", and then it seems to take the linebreak too literally after "files")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OH RIGHT, forgot it does that.

{% endhint %}


Expand All @@ -38,15 +39,19 @@ pipeline:
<...>
```

Each processor can only have a single `condition` block, but can have multiple rules within that condition.
These rules are stored as items in the `condition.rules` array.
Each processor can only have a single `condition` block, but that condition can
include multiple rules. These rules are stored as items in the `condition.rules`
array.

### Condition evaluation

The `condition.op` parameter specifies the condition's evaluation logic. It has two possible values:
The `condition.op` parameter specifies the condition's evaluation logic. It has
two possible values:

- `and`: All rules in the `condition.rules` array must evaluate to `true` for the condition to be met.
- `or`: One or more rules in the `conditions.rules` array must evaluate to `true` for the condition to be met.
- `and`: A log entry meets this condition when all of the rules in the `condition.rules`
are [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy).
- `or`: A log entry meets this condition when one or more rules in the `condition.rules`
array are [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy).

### Rules

Expand All @@ -56,7 +61,7 @@ Each item in the `condition.rules` array must include values for the following p
| --- | --- |
| `field` | The field within your logs to evaluate. The value of this parameter must use [the correct syntax](#field-access) to access the fields inside logs. |
| `op` | The [comparison operator](#comparison-operators) to evaluate whether the rule is true. This parameter (`condition.rules.op`) is distinct from the `condition.op` parameter and has different possible values. |
| `value` | The value of the specified log field to use in your comparison. Optionally, you can provide [an array that contains multiple values](#array-of-values). |
| `value` | The value of the specified log field to use in your comparison. Optionally, you can provide [an array that contains multiple values](#array-of-values). |

Rules are evaluated against each log that passes through your data pipeline. For example, given a rule with these parameters:

Expand Down Expand Up @@ -93,7 +98,8 @@ The `conditions.rules.op` parameter has the following possible values:

### Basic condition

This example applies a condition that only processes logs that contain the string `{"request": {"method": "POST"`:
This example applies a condition that only processes logs that contain the
string `{"request": {"method": "POST"`:

```yaml
pipeline:
Expand All @@ -117,7 +123,8 @@ pipeline:

### Multiple conditions with `and`

This example applies a condition that only processes logs when all of the specified rules are met:
This example applies a condition that only processes logs when all of the
specified rules are met:

```yaml
pipeline:
Expand All @@ -144,7 +151,8 @@ pipeline:

### Multiple conditions with `or`

This example applies a condition that only processes logs when one or more of the specified rules are met:
This example applies a condition that only processes logs when one or more of
the specified rules are met:

```yaml
pipeline:
Expand Down Expand Up @@ -234,4 +242,5 @@ pipeline:
value: ["error", "fatal"]
```

This configuration would add the `alert` field to error logs from critical services, and add the `paging_required` field to errors containing specific critical patterns.
This configuration adds an `alert` field to error logs from critical services,
and adds a `paging_required` field to errors that contain specific critical patterns.