stackdriver: Refactor special fields docs #1585

braydonk · 2025-03-10T17:09:52Z

The special fields documentation as it existed was a bit of a mishmash of old documentation for a previous logging agent solution and a lot of additions over the years trying to match the old format. It ends up reproducing a lot of information that is better stated by the existing Cloud Logging docs, while missing crucial details about how the plugin handled these cases. Certain fields were also missing.

This PR rewrites the documentation, doing more linking out to Google Cloud Logging documentation in place of rewording the same information. It also guides more generally on some of the edge cases for how the special fields are handled, calling out the exceptions in a clearer way. It also adds some fields that were previously missing from this document.

The special fields documentation as it existed was a bit of a mishmash of old documentation for a previous logging agent solution and a lot of additions over the years trying to match the old format. It ends up reproducing a lot of information that is better stated by the existing Cloud Logging docs, while missing crucial details about how the plugin handled these cases. Certain fields were also missing. This PR rewrites the documentation, doing more linking out to Google Cloud Logging documentation in place of rewording the same information. It also guides more generally on some of the edge cases for how the special fields are handled, calling out the exceptions in a clearer way. It also adds some fields that were previously missing from this document. Signed-off-by: braydonk <[email protected]>

ensonic · 2025-03-10T17:35:58Z

pipeline/outputs/stackdriver_special_fields.md

-| timestampSecond & timestampNanos | timestamp | The seconds and nanos that represents the time |
+| JSON log field | [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) field | type | Description |
+| :--- | :--- | :--- | :--- |
+| `logging.googleapis.com/logName` | `logName` | `string` | The log name to write this log to. |


Should we mention here or in this doc that each of those fields must no exceed a cardinality of 30k (https://cloud.google.com/logging/docs/logs-based-metrics/troubleshooting#too-many-time-series) if one wants to use log based metrics?

The particular way that log name happens to interact with log based metrics strikes me as something Fluent Bit itself shouldn't be concerned about, so I wasn't sure the right way to bring that in. I think I found a nice way to bring it in, while still mostly leaning on linking out to our own docs on the subjects. In 95ce2ea I add a new section to the main stackdriver plugin docs about default Log Name behaviour, along with how to customize it and some examples for why you may want to do this. Hopefully this should get across the point for anyone else who may encounter the same problem.

Side note: I think our own docs on this aren't amazing on this front either, and could probably do with some polishing. I'll raise it with the technical writers who (I think might) work on this internally.

Thanks a lot! An extra section makes a lot of sense.

esmerel

lots of good updates here! I'm only not approving it because of the one spot where it looks like a URL is missing and I don't want to see broken render or bad links =)

esmerel · 2025-03-10T22:07:55Z

pipeline/outputs/stackdriver.md

@@ -55,7 +55,7 @@ If you are using a _Google Cloud Credentials File_, the following configuration

 Example configuration file for k8s resource type:

-local_resource_id is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.
+`local_resource_id` is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.


Suggested change

`local_resource_id` is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.

`local_resource_id` is used by the Stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the `local_resource_id` field in the log entry. If there is no field `logging.googleapis.com/local_resource_id` in the log, the plugin will then construct it by using the tag value of the log.

Fixed in f2f67cf

There are likely a number of little cleanups to do here that I will try and find time to do. Nobody who has worked on these docs from the Google side is a technical writer by trade (nor am I) so a lot of it could be cleaner. Fixing these things ends up being sadly a bit of a side priority, but I will still try and come back and clean stuff up when the opportunity presents itself.

esmerel · 2025-03-10T22:09:48Z

pipeline/outputs/stackdriver_special_fields.md


-The JSON representation is as followed:
+**If the field itself is not a map, the plugin will leave this field untouched.** For example:


Suggested change

**If the field itself is not a map, the plugin will leave this field untouched.** For example:

If the field itself is not a map, the plugin will leave this field untouched. For example:

Fixed in 95ce2ea

esmerel · 2025-03-10T22:10:16Z

pipeline/outputs/stackdriver_special_fields.md

-HttpRequest field is a common proto for logging HTTP requests.
-
-The JSON representation is as followed:
+**If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload.** For example:


Suggested change

**If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload.** For example:

If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload. For example:

Fixed in 95ce2ea

esmerel · 2025-03-10T22:12:15Z

pipeline/outputs/stackdriver_special_fields.md


-The span ID within the trace associated with the log entry.
+The `labels` field is expected to be an `object<string, string>`. If any fields have a value that is not a string, the value will be ignored and **not preserved**. The plugin will log an error and drop the field.


Suggested change

The `labels` field is expected to be an `object<string, string>`. If any fields have a value that is not a string, the value will be ignored and **not preserved**. The plugin will log an error and drop the field.

The `labels` field is expected to be an `object<string, string>`. If any fields have a value that is not a string, the value will be ignored and not preserved. The plugin logs an error and drops the field.

Fixed in f2f67cf

braydonk · 2025-03-12T19:44:48Z

the one spot where it looks like a URL is missing

Oops! Thanks, fixed in 40631d3.

braydonk · 2025-03-12T19:48:50Z

Ah oops, I don't have automatic commit signing set up in my local environment for this repo. Force-push coming to fix the signing sorry!

Signed-off-by: braydonk <[email protected]>

braydonk · 2025-03-12T20:01:10Z

All open feedback should be addressed now, thanks!

braydonk requested review from a team as code owners March 10, 2025 17:09

braydonk mentioned this pull request Mar 10, 2025

Stackdriver logs have high cardinality logName fields fluent/fluent-bit#9897

Open

ensonic reviewed Mar 10, 2025

View reviewed changes

esmerel reviewed Mar 10, 2025

View reviewed changes

braydonk force-pushed the sd-refactor-special-fields branch from df54b1f to feed5e5 Compare March 12, 2025 19:55

braydonk added 2 commits March 12, 2025 19:58

address review feedback

95ce2ea

Signed-off-by: braydonk <[email protected]>

address additional review feedback

f2f67cf

Signed-off-by: braydonk <[email protected]>

braydonk force-pushed the sd-refactor-special-fields branch from feed5e5 to f2f67cf Compare March 12, 2025 19:58

braydonk requested review from ensonic and esmerel March 12, 2025 20:01

patrick-stephens approved these changes Mar 13, 2025

View reviewed changes

patrick-stephens merged commit 096aab5 into fluent:master Mar 13, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stackdriver: Refactor special fields docs #1585

stackdriver: Refactor special fields docs #1585

braydonk commented Mar 10, 2025

ensonic Mar 10, 2025

braydonk Mar 12, 2025 •

edited

Loading

ensonic Mar 13, 2025

esmerel left a comment

esmerel Mar 10, 2025

braydonk Mar 12, 2025

esmerel Mar 10, 2025

braydonk Mar 12, 2025 •

edited

Loading

esmerel Mar 10, 2025

braydonk Mar 12, 2025 •

edited

Loading

esmerel Mar 10, 2025

braydonk Mar 12, 2025

braydonk commented Mar 12, 2025

braydonk commented Mar 12, 2025

braydonk commented Mar 12, 2025

	`local_resource_id` is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.
	`local_resource_id` is used by the Stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the `local_resource_id` field in the log entry. If there is no field `logging.googleapis.com/local_resource_id` in the log, the plugin will then construct it by using the tag value of the log.


		The JSON representation is as followed:
		If the field itself is not a map, the plugin will leave this field untouched. For example:

	If the field itself is not a map, the plugin will leave this field untouched. For example:
	If the field itself is not a map, the plugin will leave this field untouched. For example:

	If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload. For example:
	If there are extra subfields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra subfields in jsonPayload. For example:


		The span ID within the trace associated with the log entry.
		The `labels` field is expected to be an `object<string, string>`. If any fields have a value that is not a string, the value will be ignored and not preserved. The plugin will log an error and drop the field.

stackdriver: Refactor special fields docs #1585

stackdriver: Refactor special fields docs #1585

Conversation

braydonk commented Mar 10, 2025

Choose a reason for hiding this comment

braydonk Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esmerel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braydonk Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braydonk Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braydonk commented Mar 12, 2025

braydonk commented Mar 12, 2025

braydonk commented Mar 12, 2025

braydonk Mar 12, 2025 •

edited

Loading

braydonk Mar 12, 2025 •

edited

Loading

braydonk Mar 12, 2025 •

edited

Loading