Skip to content

Commit 105bf16

Browse files
agup006gitbook-bot
agup006
authored and
gitbook-bot
committed
GITBOOK-7: New Sandbox and Lab Resources Page
1 parent 9642fb4 commit 105bf16

File tree

6 files changed

+92
-34
lines changed

6 files changed

+92
-34
lines changed

‎.gitbook/assets/image (1).png

1.09 MB
Loading

‎.gitbook/assets/image.png

438 KB
Loading

‎.gitbook/includes/untitled.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Untitled
3+
---
4+
5+
{% embed url="https://o11y-workshops.gitlab.io/workshop-fluentbit/lab01.html" fullWidth="false" %}
6+
Lab 1 - Introduction to Fluent Bit
7+
{% endembed %}

‎SUMMARY.md

+1
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
* [A Brief History of Fluent Bit](about/history.md)
99
* [Fluentd & Fluent Bit](about/fluentd-and-fluent-bit.md)
1010
* [License](about/license.md)
11+
* [Sandbox and Lab Resources](about/sandbox-and-lab-resources.md)
1112

1213
## Concepts
1314

‎about/sandbox-and-lab-resources.md

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
description: >-
3+
The following page gives an overview of free public resources for Sandbox and
4+
Labs for learning how to best operate, use, and have success with Fluent Bit.
5+
---
6+
7+
# Sandbox and Lab Resources
8+
9+
## Fluent Bit Sandbox - sign-up required
10+
11+
The following are labs that can run in your browser however require email sign-up
12+
13+
<div data-full-width="true"><figure><img src="../.gitbook/assets/image (1).png" alt=""><figcaption></figcaption></figure></div>
14+
15+
* [Fluent Bit 101 Sandbox - Getting Started with configuration and routing](https://play.instruqt.com/Fluent/invite/nuys5ifhsprt)
16+
17+
## Open Source Labs - environment required
18+
19+
The following are open source labs where you will need to spin up resources to run through the lab in details
20+
21+
### O11y Workshops by Chronosphere
22+
23+
These workshops, open source, provided by Chronosphere can be found here: [https://o11y-workshops.gitlab.io/](https://o11y-workshops.gitlab.io/). The OSS repository can be found here: [https://gitlab.com/o11y-workshops/workshop-fluentbit](https://gitlab.com/o11y-workshops/workshop-fluentbit)
24+
25+
The cards below include links to each of the labs in the workshop&#x20;
26+
27+
{% embed url="https://o11y-workshops.gitlab.io/workshop-fluentbit/" %}
28+
Fluent Bit Workshop for Getting Started with Cloud Native Telemetry Pipelines
29+
{% endembed %}
30+
31+
1. [Lab 1 - Introduction to Fluent Bit](https://o11y-workshops.gitlab.io/workshop-fluentbit/lab01.html)
32+
2. [Lab 2 - Installing Fluent Bit](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/4)
33+
3. [Lab 3 - Exploring First Pipelines](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/5)
34+
4. [Lab 4 - Exploring More Pipelines](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/6)
35+
5. [Lab 5 - Understanding Backpressure](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/7)
36+
6. [Lab 6 - Avoid Telemetry Data Loss](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/8)
37+
7. [Lab 7 - Pipeline Integration with OpenTelemetry](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/9)
38+
39+
### Logging with Fluent Bit and Amazon OpenSearch workshop by Amazon
40+
41+
This workshop by Amazon goes through common Kubernetes logging patterns and routing data to OpenSearch and visualizing with OpenSearch dashboards
42+
43+
{% embed url="https://archive.eksworkshop.com/intermediate/230_logging/" %}

‎pipeline/filters/log_to_metrics.md

+41-34
Original file line numberDiff line numberDiff line change
@@ -2,40 +2,41 @@
22
description: Generate metrics from logs
33
---
44

5-
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=768830f6-8d2d-4231-9e5e-259ce6797ba5" />
5+
# Log to Metrics
66

7-
# Log To Metrics
7+
![](https://static.scarf.sh/a.png?x-pxid=768830f6-8d2d-4231-9e5e-259ce6797ba5)
8+
9+
## Log To Metrics
810

911
The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to count records, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline.
1012

1113
_Please note that this plugin is an experimental feature and is not recommended for production use. Configuration parameters and plugin functionality are subject to change without notice._
1214

13-
14-
## Configuration Parameters
15+
### Configuration Parameters
1516

1617
The plugin supports the following configuration parameters:
1718

18-
| Key | Description | Mandatory | Value Format
19-
| :--- | :--- | :--- | :---
20-
| tag | Defines the tag for the generated metrics record| Yes | |
21-
| metric_mode | Defines the mode for the metric. Valid values are [`counter`, `gauge` or `histogram`] | Yes | |
22-
| metric_name | Sets the name of the metric. | Yes | |
23-
| metric_description | Sets a help text for the metric. | Yes | |
24-
| bucket | Defines a bucket for `histogram` | Yes, for mode `histogram` | e.g. 0.75 |
25-
| add_label | Add a custom label NAME and set the value to the value of KEY | | | NAME KEY |
26-
| label_field | Includes a record field as label dimension in the metric. | | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields.
27-
| value_field | Specify the record field that holds a numerical value | Yes, for modes [`gauge` and `histogram`] | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields.
28-
| kubernetes_mode | If enabled, it will automatically put pod_id, pod_name, namespace_name, docker_id and container_name into the metric as labels. This option is intended to be used in combination with the [kubernetes](./kubernetes.md) filter plugin, which fills those fields. | |
29-
| Regex | Include records in which the content of KEY matches the regular expression. | | KEY REGEX
30-
| Exclude | Exclude records in which the content of KEY matches the regular expression. | | KEY REGEX
31-
| Flush\_Interval\_Sec | The interval for metrics emission, in seconds. If **Flush\_Interval\_Sec** and **Flush\_Interval\_Nsec** are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. Default value: `0`. |
32-
| Flush\_Interval\_Nsec | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with **Flush\_Interval\_Sec**. Default value: `0`. |
33-
34-
## Getting Started
19+
| Key | Description | Mandatory | Value Format |
20+
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
21+
| tag | Defines the tag for the generated metrics record | Yes | |
22+
| metric\_mode | Defines the mode for the metric. Valid values are \[`counter`, `gauge` or `histogram`] | Yes | |
23+
| metric\_name | Sets the name of the metric. | Yes | |
24+
| metric\_description | Sets a help text for the metric. | Yes | |
25+
| bucket | Defines a bucket for `histogram` | Yes, for mode `histogram` | e.g. 0.75 |
26+
| add\_label | Add a custom label NAME and set the value to the value of KEY | | |
27+
| label\_field | Includes a record field as label dimension in the metric. | | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
28+
| value\_field | Specify the record field that holds a numerical value | Yes, for modes \[`gauge` and `histogram`] | Name of record key. Supports [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
29+
| kubernetes\_mode | If enabled, it will automatically put pod\_id, pod\_name, namespace\_name, docker\_id and container\_name into the metric as labels. This option is intended to be used in combination with the [kubernetes](kubernetes.md) filter plugin, which fills those fields. | | |
30+
| Regex | Include records in which the content of KEY matches the regular expression. | | KEY REGEX |
31+
| Exclude | Exclude records in which the content of KEY matches the regular expression. | | KEY REGEX |
32+
| Flush\_Interval\_Sec | The interval for metrics emission, in seconds. If **Flush\_Interval\_Sec** and **Flush\_Interval\_Nsec** are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. Default value: `0`. | | |
33+
| Flush\_Interval\_Nsec | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with **Flush\_Interval\_Sec**. Default value: `0`. | | |
34+
35+
### Getting Started
3536

3637
The following example takes records from two dummy inputs and counts all messages passing through the `log_to_metrics` filter. It then generates metric records which are provided to the `prometheus_exporter`:
3738

38-
### Configuration - Counter
39+
#### Configuration - Counter
3940

4041
```python
4142
[SERVICE]
@@ -68,7 +69,8 @@ The following example takes records from two dummy inputs and counts all message
6869
```
6970

7071
You can then use e.g. curl command to retrieve the generated metric:
71-
```text
72+
73+
```
7274
> curl -s http://127.0.0.1:2021/metrics
7375
7476
@@ -77,9 +79,10 @@ You can then use e.g. curl command to retrieve the generated metric:
7779
log_metric_counter_count_all_dummy_messages 49
7880
```
7981

80-
### Configuration - Gauge
82+
#### Configuration - Gauge
8183

8284
The `gauge` mode needs a `value_field` specified, where the current metric values are generated from. In this example we also apply a regex filter and enable the `kubernetes_mode` option:
85+
8386
```python
8487
[FILTER]
8588
name log_to_metrics
@@ -95,8 +98,10 @@ The `gauge` mode needs a `value_field` specified, where the current metric value
9598
label_field color
9699
label_field shape
97100
```
101+
98102
You can then use e.g. curl command to retrieve the generated metric:
99-
```text
103+
104+
```
100105
> curl -s http://127.0.0.1:2021/metrics
101106
102107
@@ -107,16 +112,17 @@ log_metric_gauge_current_duration{namespace_name="default",pod_name="pod1",conta
107112

108113
As you can see in the output, only one line is printed, as the records from the first input plugin are ignored, as they do not match the regex.
109114

110-
The filter also allows to use multiple rules which are applied in order, you can have many _Regex_ and _Exclude_ entries as required (see [grep](./grep.md) filter plugin).
115+
The filter also allows to use multiple rules which are applied in order, you can have many _Regex_ and _Exclude_ entries as required (see [grep](grep.md) filter plugin).
111116

112117
If you execute the above `curl` command multiple times, you see, that in this example the metric value stays at `60`, as the messages generated by the dummy plugin are not changing. In a real-world scenario the values would change and return the last processed value.
113118

119+
**Metric label\_values**
114120

115-
#### Metric label_values
116121
As you can see, the label sets defined by `add_label` and `label_field` are added to the metric. The lines in the metric represent every combination of labels. Only actually used combinations are displayed here. To see this, you can add a dummy `dummy` input to your configuration.
117122

118123
The metric output would then look like:
119-
```text
124+
125+
```
120126
> curl -s http://127.0.0.1:2021/metrics
121127
122128
# HELP log_metric_gauge_current_duration This metric shows the current duration
@@ -126,11 +132,12 @@ log_metric_gauge_current_duration{namespace_name="default",pod_name="pod1",conta
126132
127133
```
128134

129-
You can also see, that all the kubernetes labels have been attached to the metric, accordingly.
135+
You can also see, that all the kubernetes labels have been attached to the metric, accordingly.
130136

131-
### Configuration - Histogram
137+
#### Configuration - Histogram
132138

133139
Similar to the `gauge` mode, `histogram` needs a `value_field` specified, where the current metric values are generated from. In this example we also apply a regex filter and enable the `kubernetes_mode` option:
140+
134141
```python
135142
[FILTER]
136143
name log_to_metrics
@@ -146,8 +153,10 @@ Similar to the `gauge` mode, `histogram` needs a `value_field` specified, where
146153
label_field color
147154
label_field shape
148155
```
156+
149157
You can then use e.g. curl command to retrieve the generated metric:
150-
```text
158+
159+
```
151160
> curl -s http://127.0.0.1:2021/metrics
152161
153162
@@ -210,6 +219,4 @@ As you can see in the output, there are per default the buckets `0.005, 0.01, 0.
210219

211220
Please note, that the `+Inf` bucket will always be included implicitly. The buckets in a histogram are cumulative, so a value added to one bucket will add to all larger buckets, too.
212221

213-
214-
This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`.
215-
This results in two sets for the histogram.
222+
This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram.

0 commit comments

Comments
 (0)