Add telemetry to quickstart

Author: doodlesbykumbi

README.md

@@ -14,6 +14,7 @@ using Docker Compose.
     + [Run the demo app](#run-the-demo-app)
   * [Next steps](#next-steps)
   * [Explore the Conjur database](#explore-the-conjur-database)
+  * [Use Conjur with telemetry](#use-conjur-with-telemetry)
   * [Configuring Conjur with predefined admin password](#configuring-conjur-with-predefined-admin-password)
   * [Using persistent Conjur configuration](#using-persistent-conjur-configuration)
     + [Set up a Conjur Open Source environment with persistence](#set-up-a-conjur-open-source-environment-with-persistence)
@@ -357,6 +358,60 @@ To explore the database
 
 ![image](https://user-images.githubusercontent.com/8653164/115864622-03da7a00-a42f-11eb-974f-dc2cb034ca09.png)
 
+### Use Conjur with telemetry
+
+Conjur supports telemetry as an opt-in feature. The telemetry feature has a general purpose mechanism for collection, but currently only supports a single method for export, a Prometheus scrape target endpoint. Below are instructions for enabling and exploring the telemetry feature.
+
+In order to enable telemetry in Conjur you must opt-in via configuration.
+You have a choice between setting an environment variable:
+
+```sh
+CONJUR_TELEMETRY_ENABLED=true
+```
+
+or updating a value in the `conjur.conf` configuration file:
+
+```yaml
+telemetry_enabled: true
+```
+
+Note that the environment variables takes precedence.
+
+1. If you are already running the Conjur Open Source quickstart environment without
+   telemetry, bring down the Conjur container:
+
+   ```
+   docker-compose down conjur
+   ```
+
+1. Modify `docker-compose.yml` in this repository to enable telemetry
+   by setting the `CONJUR_TELEMETRY_ENABLED` environment variable to the value `'true'` (It needs to be a string otherwise the docker-compose YAML parser will not be happy). Below is an illustration of the required change:
+
+   ```yaml
+   services:
+      # ...
+      conjur:
+         environment:
+            # ...
+            CONJUR_TELEMETRY_ENABLED: 'true'
+   ```
+
+1. Start the Conjur Open Source environment using telemetry:
+
+   - If you had previously been running the Conjur Open Source environment,
+     follow the steps 2 and 3 of the
+     [Set up a Conjur Open Source environment](#set-up-a-conjur-open-source-environment)
+     section above in order to recreate the Conjur container.
+   - Otherwise, follow the steps starting from Step 1 of the
+     [Set up a Conjur Open Source environment](#set-up-a-conjur-open-source-environment)
+     section above.
+
+1. Navigate to the [telemetry README](./telemetry/README.md#getting-started) and, starting from step 2, follow the instructions to set up the telemetry related services.
+
+   The telemetry README provides instructions for a comprehensive quickstart for setting up services such as Prometheus and Grafana,
+   creating relevant connections between those services and the Conjur Prometheus scrape target endpoint, and providing an example
+   dashboard with the metrics collected by Conjur.
+
 ### Configuring Conjur with predefined admin password
 
 The following command will allow you to specify the admin user's password:

docker-compose.yml

@@ -58,6 +58,7 @@ services:
       DATABASE_URL: postgres://postgres:SuperSecretPg@database/postgres
       CONJUR_DATA_KEY:
       CONJUR_AUTHENTICATORS:
+      CONJUR_TELEMETRY_ENABLED: 'false'
     depends_on:
     - database
     restart: on-failure

telemetry/README.md

@@ -0,0 +1,75 @@
+# Conjur telemetry quickstart
+
+In order to consume the telemetry data collected by Conjur, you need to setup a Prometheus node to scrape the
+`/metrics` endpoint in Conjur. You can then use the myriad of tools that work on top of Prometheus to visualise the data. 
+
+Here, we provide a quickstart that sets up Prometheus and Grafana to allow you to quickly explore visualisation of the telemetry data collected by Conjur. To be comprehensive in our demostration, this quickstart additionally provides examples of gathering of metrics from services adjacent to Conjur such as Postgres and metrics about the node where Conjur is running. 
+
+This quickstart auto-magically carries out the following important tasks:
+1. Configure Prometheus to consume metrics from the 3 exporters (node, postgres and conjur).
+1. Configure Grafana to use Prometheus as a data source. 
+1. Create an example dashboard in Grafana that brings together all these metrics.
+
+Once the services are set-up, all the metrics will available within Grafana for visual consumption.
+
+The "important" tasks listed above are really just carried out by providing configuration files, which can be found under [./files](./files/).
+
+## Getting Started 
+
+1. First, follow the instructions of the [README.md in the root directory of the repository](../README.md#use-conjur-with-telemetry) to
+   setup Conjur with telemetry. Once completed, those instructions will ask you to come back here and continue with the next step.
+
+1. Pull the Docker images
+
+   Open a terminal session and change directory to the folder containg this README. Pull the Docker
+   images defined in `docker-compose.yml`:
+   ```
+   docker-compose pull
+   ```
+
+   **Verification**
+   When the required images are successfully pulled, the terminal returns the
+   following:
+   ```
+   [+] Running 4/4
+   ⠿ node-exporter Pulled                                                                                           1.4s
+   ⠿ prometheus Pulled                                                                                              1.4s
+   ⠿ pg-prometheus-exporter Pulled                                                                                  1.4s
+   ⠿ grafana Pulled                                                                                                 1.4s
+   ```
+
+1. Start all the services. The services are the Prometheus exporters for postgres and the node (pg-prometheus-exporter and
+   node-exporter), prometheus and grafana. 
+
+   Start the services:
+   ```
+   docker-compose up -d
+   ```
+
+   When the services start the terminal returns the following:
+   ```
+   [+] Running 4/0
+   ⠿ Container telemetry-grafana-1                 Running                                                          0.0s
+   ⠿ Container telemetry-node-exporter-1           Running                                                          0.0s
+   ⠿ Container telemetry-pg-prometheus-exporter-1  Running                                                          0.0s
+   ⠿ Container telemetry-prometheus-1              Running                                                          0.0s
+   ```
+
+   **Verification**
+   Run the following command to see a list of running containers:
+   ```
+   docker-compose ps
+   ```
+
+1. Use the Grafana UI
+
+   1. On a local browser, navigate to the Grafana UI available at [http://localhost:2345](http://localhost:2345).
+   1. Log in as `admin`/`admin`. You'll be prompted to change password, but for evaluation purposes you have the option to skip that step.
+   1. Once logged in, on the left column, select Dashboards. Under `General` in the `Browse` tab , select `Conjur Dashboard` from the list of available dashboards. See image below.
+
+      ![](./select-dashboard.png)
+   1. The page should now be populated with the Conjur dashboard. See image below.
+
+      ![](./dashboard.png)
+   1. Use Conjur as usual and observe how your activity is reflected in the updates to the Conjur dashboard.
+

telemetry/dashboard.png

@@ -0,0 +1,37 @@
+version: '3'
+services:
+  pg-prometheus-exporter:
+    image: quay.io/prometheuscommunity/postgres-exporter
+    environment: 
+      DATA_SOURCE_NAME: postgres://postgres@database/postgres?sslmode=disable
+      PG_EXPORTER_CONSTANT_LABELS: component=pg
+
+  prometheus:
+    image: prom/prometheus
+    volumes:
+      - ./files/prometheus:/etc/prometheus
+    ports:
+      - 9090:9090
+    command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
+
+  grafana:
+    image: grafana/grafana
+    environment:
+      - GF_SERVER_HTTP_PORT=2345
+    ports:
+      - 2345:2345
+    volumes:
+      - ./files/grafana/storage:/var/lib/grafana
+      - ./files/grafana/grafana.ini:/etc/grafana/grafana.ini
+      - ./files/grafana/provisioning:/etc/grafana/provisioning
+      - ./files/grafana/dashboards:/etc/grafana/dashboards
+
+  # Node exporter provides CPU and Memory metrics to Prometheus for the Docker
+  # host machine.
+  node-exporter:
+    image: quay.io/prometheus/node-exporter:latest
+
+networks:
+  default:
+    name: conjur-quickstart_default
+    external: true