diff --git a/advocacy_docs/pg_extensions/extensionrefs.json b/advocacy_docs/pg_extensions/extensionrefs.json index 111bbbb5fc1..fda5f49b0ec 100644 --- a/advocacy_docs/pg_extensions/extensionrefs.json +++ b/advocacy_docs/pg_extensions/extensionrefs.json @@ -128,5 +128,6 @@ "plv8":"https://github.com/plv8/plv8", "aidb":"https://www.enterprisedb.com/docs/edb-postgres-ai/ai-accelerator/pipelines-overview/", "pgfs":"https://www.enterprisedb.com/docs/edb-postgres-ai/ai-accelerator/pgfs/", - "bluefin":"https://www.enterprisedb.com/docs/pg_extensions/advanced_storage_pack/" + "bluefin":"https://www.enterprisedb.com/docs/pg_extensions/advanced_storage_pack/", + "edb_otel": "https://www.enterprisedb.com/docs/pg_extensions/otel/" } \ No newline at end of file diff --git a/advocacy_docs/pg_extensions/index.mdx b/advocacy_docs/pg_extensions/index.mdx index bc57743e34f..d4ac7194191 100644 --- a/advocacy_docs/pg_extensions/index.mdx +++ b/advocacy_docs/pg_extensions/index.mdx @@ -18,9 +18,11 @@ navigation: - sqlprofiler - system_stats - wal2json + - otel - "#EDB Postgres Advanced Server only" - spl_check - edb_job_scheduler + --- Categories of extensions: @@ -138,7 +140,8 @@ Categories of extensions: - + + @@ -162,8 +165,8 @@ Categories of extensions: - - + +
Virtual Machines/Physical ServersKubernetesEDB Postgres AI Cloud Service
Extension nameRequires
superuser
access		PostgreSQLEDB
Postgres
Extended
Server		EDB
Postgres
Advanced
Server		PostgreSQLEDB
Postgres
Advanced
Server		PostgreSQLEDB
Postgres
Extended
Server		EDB
Postgres
Advanced
Server
edb_pg_tuner✓✓✓✓✓–✓✓
Extension nameRequires
superuser
access		PostgreSQLEDB
Postgres
Extended
Server		EDB
Postgres
Advanced
Server		PostgreSQLEDB
Postgres
Advanced
Server		PostgreSQLEDB
Postgres
Extended
Server		EDB
Postgres
Advanced
Server
edb_otel✓✓✓–––––
edb_pg_tuner✓✓✓✓✓–✓✓
edb_query_advisor✓✓✓––✓✓✓
edb_wait_states✓✓✓––✓✓✓
sql_profiler✓✓✓––✓✓✓
bdr 5.x✓✓✓PreviewPreview–PreviewPreview
pglogical 3.xYes✓✓✓–––––
AI Accelerator Pipelines
aidb✓✓✓––✓✓✓
pgfs✓✓✓––✓✓✓
aidbYes✓✓✓––✓✓✓
pgfsYes✓✓✓––✓✓✓
diff --git a/advocacy_docs/pg_extensions/index.mdx.in b/advocacy_docs/pg_extensions/index.mdx.in index 8af7d6d7f62..92edbfc3b0f 100644 --- a/advocacy_docs/pg_extensions/index.mdx.in +++ b/advocacy_docs/pg_extensions/index.mdx.in @@ -18,6 +18,7 @@ navigation: - sqlprofiler - system_stats - wal2json + - otel - "#EDB Postgres Advanced Server only" - spl_check - edb_job_scheduler diff --git a/advocacy_docs/pg_extensions/otel/configuring.mdx b/advocacy_docs/pg_extensions/otel/configuring.mdx new file mode 100644 index 00000000000..75bf2a36bee --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/configuring.mdx @@ -0,0 +1,88 @@ +--- +title: Configuring EDB OTEL +navTitle: Configuring +--- + +The following configuration options are available with edb_otel. + +## Permissions + +No permissions changes are required for edb_otel. + +## Prerequisites + +Before using edb_otel to export metrics, you must set up an OpenTelemetry collector endpoint that edb_otel can communicate with. For detailed information on how to set up a quick test of the Opentelemetry endpoint using docker, see [OTLP Exporter Example](https://github.com/open-telemetry/opentelemetry-cpp/tree/main/examples/otlp). + +## GUCs + +The following GUC variables are available in edb_otel: + +| Variable | Description | Unit | Default | Minimum value | | +|--------------------------------|-------------|------|---------|---------------| +| `edb_otel.edb_otel_endpoint` | Specifies the OTEL endpoint URL. | | | | +| `edb_otel.interval` | Interval between two consecutive exports. It translates to the opentelemetry metric reader parameter `exportIntervalMillis`.| Milliseconds | 2 seconds | 1 second| +| `edb_otel.timeout` | Specifies how long the export can run before it's cancelled. It translates to the opentelemetry metric reader parameter `exportTimeoutMillis`.| Milliseconds | 500 milliseconds | 100 milliseconds| +| `edb_otel.worker_nap_time` | Duration of background worker sleep time before it checks for metrics in the queue and processes them. | Milliseconds | 2 seconds | 1 second | +| `edb_otel.metrics_queue_size` | Interval between two consecutive exports. This is the maximum size of the metrics queue. If the queue reaches this size, any subsequent metrics are dropped until the queue is cleared while processing the metrics inside it.| MB | 100 | 100| + +## Configuring to be used by an SQL interface + +To use the edb_otel extension from a SQL interface, use the following steps: + +In `postgresql.conf`: + +1. Set the parameter `edb_otel.edb_otel_endpoint` to point to the collector endpoint. Example: +```bash +edb_otel.edb_otel_endpoint='localhost:4317' +``` +2. Add `edb_otel` to the `shared_preload_libraries`. Example: +```bash +shared_preload_libraries = '$libdir/dbms_pipe,$libdir/edb_gen,$libdir/dbms_aq,$libdir/edb_otel' +``` +## Configuring to be used within C code + +To use the edb_otel extension from your C code, use the following steps: + +1. Ensure the `edb_otel_ext.h` file is within reach of your extension at build time. The file is placed in `$prefix/include/server/extension/edb_otel/` when the edb_otel dev/devel package is installed. Your Makefile must add `pg_config --includedir-server` to its include search paths, so your extension can use it as: +```C +#include "extension/edb_otel/edb_otel_ext.h" +``` +This header contains the signature of the function that is used to send metrics to the OTEL endpoint. + +2. When the edb_otel extension is loaded, it places a function pointer in the rendezvous variable identified by `edb_otel_metric_func`. So the next step is to get that function pointer: +```C +edb_otel_metric_func_type * p_edb_otel_metric = + (edb_otel_metric_func_type *) find_rendezvous_variable("edb_otel_metric_func"); +``` +The function pointer may be stored in a session-long variable, but care should be taken in case edb_otel gets loaded after the calling code. + +Since the instrumented code must continue to work even when edb_otel isn't in use, if the function pointer returns NULL, no error should be raised, and the function must not be called. + +For example, instrumenting pg_stat_statements to send the number of plans may look similar to this: + +```C +if (*p_edb_otel_metric) { + char val[100] = {0}; + sprintf(val, "%ld", e->counters.calls[PGSS_PLAN]); + (*p_edb_otel_metric)("pg_stat_statements", + "plans", + EDB_OTEL_GAUGE, + val, + labels); +} +``` + +The `edb_otel_metric` function has the following parameters: + +| Parameter(s) | Input or output | Description | +|--------------------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `meter_name` | Input | A name to identify a group of metrics. | +| `metric_name` | Input | A name to identify an actual measurement of the metric. | +| `metric_type` | Input | Identified by one of the enums in `edb_otel_ext.h`.| +| `measurement` | Input | A string value.| +| `labels` | Input | A jsonb string representing additional metadata to be associated with the metric, for example, attaching the query text to the measurement of `pg_stat_statements`.| + +!!!Note +Care should be taken when calling the function that sends metrics. It's important to avoid using it in hot paths and while holding locks, since it currently enters third party code. In the future, it may still perform non-trivial work, if only pushing data through a shm_mq to a background worker. +!!! +For a detailed example of this C API usage where `pg_stat_statements` was instrumented to send all metrics through OTEL, see this [example](https://github.com/EnterpriseDB/opentelemetry-work/blob/arthur/pg17/contrib/pg_stat_statements/pg_stat_statements.c). \ No newline at end of file diff --git a/advocacy_docs/pg_extensions/otel/index.mdx b/advocacy_docs/pg_extensions/otel/index.mdx new file mode 100644 index 00000000000..8144285b53a --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/index.mdx @@ -0,0 +1,20 @@ +--- +title: 'EDB OTEL' +indexCards: simple +navigation: + - rel_notes + - installing + - uninstalling + - configuring + - using + - upgrading +directoryDefaults: + product: EDB OTEL +--- + +EDB OTEL (edb_otel) is an extension that allows metrics to be sent through the [OpenTelemetry](https://opentelemetry.io) (OTEL) protocol. It communicates with an OTEL endpoint and sends metrics to it. Other extensions can use APIs to pass metrics to edb_otel, which will in turn redirect to the OTEL endpoint. + + +edb_otel includes a background worker process which is responsible for managing the metric instruments and exporting the metrics to the opentelemetry metrics collector. The worker handles calls to the OpenTelemetry C++ SDK, preventing them from blocking the main backend process. Metric data, reported by other backends via a shared memory queue, is periodically retrieved and processed by the worker. + +Subsequent EDB extension releases will leverage edb_otel APIs for centralized routing OTEL metrics. diff --git a/advocacy_docs/pg_extensions/otel/installing.mdx b/advocacy_docs/pg_extensions/otel/installing.mdx new file mode 100644 index 00000000000..0159f5ba3d1 --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/installing.mdx @@ -0,0 +1,92 @@ +--- +title: "Installing EDB OTEL" +navTitle: Installing +--- + +To install and enable the edb_otel extension, use the following steps. + +The edb_otel extension is only available for PostgreSQL version 17 on RHEL 9. + +## Installing the package + +### Prerequisites + +Before you begin the installation process: + +- Install Postgres. See: + - [Installing EDB Postgres Advanced Server](/epas/latest/installing/) + + - [Installing PostgreSQL](https://www.postgresql.org/download/) + + - [Installing EDB Postgres Extended Server](/pge/latest/installing/) + +- Set up the repository. + + Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step. + + To set up the repository, go to [EDB repositories](https://www.enterprisedb.com/repos-downloads) and follow the instructions provided there. + + +### Install the package + +The syntax to install edb_otel on EDB Postgres Advanced Server and EDB Postgres Extended is: + +```shell +# For EDB Postgres Advanced Server: +sudo -y install edb-as-edb_otel + +# For EDB Postgres Extended: +sudo -y install edb-postgresextended-edb_otel +``` + +Where: + +- `` is the package manager used with your operating system: + + | Package manager | Operating system | + | --------------- | -------------------------------- | + | dnf | RHEL 9 and derivatives | + + +- `` is the version of Postgres you're using. + +For example, to install the latest version of edb_otel for EDB Postgres Advanced Server 17 on a RHEL 9 platform: + +```shell +sudo dnf -y install edb-as17-edb_otel +``` + +The syntax to install edb_otel on PostgreSQL is: + +```shell +# For RHEL 9 and its derivatives: +sudo dnf -y install edb_otel_ +``` + +Where `` is the version of PostgreSQL you're using. + + +## Enabling the extension + +To enable the extension: + +1. Edit the `postgresql.conf` file, modifying the `shared_preload_libraries` parameter: + + ```ini + shared_preload_libraries = '$libdir/edb_otel' + ``` + +1. Restart the Postgres server. + +1. Create the edb_otel extension in your database with the following command: + + ```sql + # Connect to your database using psql + psql -d postgres -U enterprisedb + # Create the extension + CREATE EXTENSION edb_otel; + ``` + + + + diff --git a/advocacy_docs/pg_extensions/otel/rel_notes/edb_otel_1.0.0_rel_notes.mdx b/advocacy_docs/pg_extensions/otel/rel_notes/edb_otel_1.0.0_rel_notes.mdx new file mode 100644 index 00000000000..968a2098caf --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/rel_notes/edb_otel_1.0.0_rel_notes.mdx @@ -0,0 +1,11 @@ +--- +title: Release notes for EDB OTEL version 1.0.0 +navTitle: "Version 1.0.0" +--- +Release date: 28 March 2025 + +This release of edb_otel includes: + +| Type | Description | +|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Release | This is the initial release.| diff --git a/advocacy_docs/pg_extensions/otel/rel_notes/index.mdx b/advocacy_docs/pg_extensions/otel/rel_notes/index.mdx new file mode 100644 index 00000000000..a903fcc8967 --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/rel_notes/index.mdx @@ -0,0 +1,12 @@ +--- +title: 'EDB OTEL release notes' +navTitle: "Release notes" +indexCards: none +--- + +The edb_otel documentation describes the latest version of edb_otel, including minor releases and patches. These release notes cover what was new in each release. For new functionality introduced in a minor or patch release, there are also indicators in the content about the release that introduced the feature. + +| Version | Release Date | +| -------------------------------- | ------------ | +| [1.0.0](edb_otel_1.0.0_rel_notes) | 04 March 2025 | + diff --git a/advocacy_docs/pg_extensions/otel/uninstalling.mdx b/advocacy_docs/pg_extensions/otel/uninstalling.mdx new file mode 100644 index 00000000000..14549ea3ea9 --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/uninstalling.mdx @@ -0,0 +1,25 @@ +--- +title: "Uninstalling EDB OTEL" +navTitle: Uninstalling +--- + +To uninstall edb_otel: + +1. Drop the edb_otel extension: + + ```sql + # Connect to your database using psql + psql -d postgres -U enterprisedb + # Drop the extension + DROP EXTENSION edb_otel; + ``` + +1. Edit the `postgresql.conf` file on the server you want to remove the extension, deleting the `shared_preload_libraries` parameter line. + + In Linux, remove: + + ```ini + shared_preload_libraries = `$libdir/edb_otel` + ``` + +1. Restart the Postgres database server to apply the changes. \ No newline at end of file diff --git a/advocacy_docs/pg_extensions/otel/upgrading.mdx b/advocacy_docs/pg_extensions/otel/upgrading.mdx new file mode 100644 index 00000000000..63e5f51e8f3 --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/upgrading.mdx @@ -0,0 +1,12 @@ +--- +title: "Upgrading EDB OTEL" +navTitle: Upgrading +--- + +To upgrade edb_otel, use the `ALTER EXTENSION` command: + +For example, to upgrade from version 1.0 to version 1.1 (not yet released): +``` +# running as superuser +ALTER EXTENSION edb_otel UPDATE TO '1.1'; +``` diff --git a/advocacy_docs/pg_extensions/otel/using.mdx b/advocacy_docs/pg_extensions/otel/using.mdx new file mode 100644 index 00000000000..67e0dbb7912 --- /dev/null +++ b/advocacy_docs/pg_extensions/otel/using.mdx @@ -0,0 +1,128 @@ +--- +title: "Using EDB OTEL" +navTitle: Using +--- + +The edb_otel extension allows metrics to be sent through the OpenTelemetry (OTEL) protocol. + + +## edb_otel APIs + +The following functions are available in edb_otel: + +- edb_otel.report_metric +- edb_otel.command_from_pg_exporter_definition +- edb_otel.schedule_from_pg_exporter_definition + +The following views are available in edb_otel: + +- edb_otel.pg_exporter_base_definitions + + +## `edb_otel.report_metric` + +Use this function to define and group metrics. + +```sql +edb_otel.report_metric( + meter_name TEXT, + metric_name TEXT, + metric_type INT, + value TEXT, + labels TEXT +) +``` + +| Parameter(s) | Input or output | Description | +|--------------------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `meter_name` | Input | A name that you want to group a set of metrics against. | +| `metric_name` | Input | The name of the metric. | +| `metric_type` | Input | The type of the metric. Only the DOUBLE GAUGE type is currently supported, which is represented by the value 4. | +| `value` | Input | A string representing the metric value. It will be converted to an appropriate type depending on the `metric_type`. | +| `labels` | Input | A json string representing additional metadata to be associated with the metric. The maximum size is 1024. If the size is larger than 1024, that metric is silently dropped and a WARNING is emitted to the database log file. If the payload contains an invalid json, that metric is dropped and error information is logged to the log file.| + + +#### Example: + + ```sql + SELECT edb_otel.report_metric('m1', 'foo', 4, '10', '{"dbname": "foo", "schemaname": "bar", "relname": "baz"}'); + ``` + + +## `edb_otel.command_from_pg_exporter_definition` +Use this function to generate a command that can run the query and send the results through `edb_otel`. It accepts a jsonb value containing the same fields as `pg_exporter` definitions. + +```sql +edb_otel.command_from_pg_exporter_definition( + meter jsonb +) +``` + + +## `edb_otel.schedule_from_pg_exporter_definition` + +Use this function to define and schedule custom queries. The query and its metadata, such as metric types and tags, must use the a jsonb structure. It accepts a jsonb value containing the parameters that are usually consumed by pg_exporter, and schedules a pg_cron job that runs the query periodically and sends the results to the OpenTelemetry endpoint. + + +```sql +edb_otel.schedule_from_pg_exporter_definition( + meter jsonb, + schedule text +) +``` + +#### Example: + +This custom query adds a count of client backends to the schedule: + + ```sql + SELECT edb_otel.schedule_from_pg_exporter_definition( + meter := $${ + "name": "user_activity", + "query": "SELECT count(*) AS user_count + FROM pg_stat_activity + WHERE backend_type = 'client backend'", + "metrics": [{"user_count": {"usage": "GAUGE"}}] + }$$::jsonb, + schedule := '* * * * *'); + ``` + + +## `edb_otel.pg_exporter_base_definitions` + +This view contains the meter (queries and their metrics) definitions that `pg_exporter` brings by default. They can be refreshed from the [upm-metrics-pg-exporter](https://github.com/EnterpriseDB/upm-metrics-pg-exporter/tree/main/config/collector) or the [pg_exporter collector](https://github.com/Vonng/pg_exporter/tree/main/config/collector) by running `yq` on the directory: + +```yq +yq eval-all -o=json '. as $item ireduce ({}; . * $item)' [...]/config/collector/*.yml +``` + +## Job scheduling + +You can use pg_cron to schedule jobs for the pg_exporter. edb_otel has auxiliary functions to ease the scheduling of monitoring queries. To start using pg_exporter queries, you will need the following: + +- the meter name: this is the top key in the YAML files +- a scheduling definition: this is a schedule that pg_cron will accept + +For example, the [410-pg_activity.yml](https://github.com/Vonng/pg_exporter/blob/main/config/collector/410-pg_activity.yml) query pulls output from pg_stat_activity. To run this every minute, the cron syntax is: '* * * * *'. To create the job in pg_cron: + +```sql +SELECT edb_otel.schedule_from_pg_exporter_definition(meters->'pg_activity', '* * * * *') +FROM edb_otel.pg_exporter_base_definitions; +``` +The view `edb_otel.pg_exporter_base_definitions` contains the metadata of the pg_activity task, which is filtered by the jsonb `->` operator, and sent into the function `edb_otel.schedule_from_pg_exporter_definition`. The `edb_otel.schedule_from_pg_exporter_definition` function then uses the `edb_otel.command_from_pg_exporter_definition` function to convert the jsonb metadata into a text command, which gets scheduled in pg_cron. +The function returns the same jobid that pg_cron returns: +```bash +-[ RECORD 1 ]------------------------+--- +schedule_from_pg_exporter_definition | 29 +``` + +This can then be viewed in the pg_cron job table: +```sql +SELECT jobname, active, jobid, schedule FROM cron.job; + +-[ RECORD 1 ]--------- +jobname | pg_activity +active | t +jobid | 29 +schedule | * * * * * +``` \ No newline at end of file