API v5.1 (#67)

* 20240701 - dev NGINX App Protect 5 compiler

* 20240701 - dev NGINX App Protect 5 compiler

* 20240730 - v5.1 dev

* 2024073002 - v5.1 dev

* 2024073003 dev 5.1

* 20240731 - 5.1 dev

* 2024080201 - 5.1 dev

* 20240806 - 5.1 dev
Moved resolvers config to include files
Added per-upstream resolver support
FEATURES updated

* 2024080602 - 5.1 dev
Moved HTTP upstreams to separate include files
Bugfixes
Updated postman collection

* 20240807 - 5.1 dev
Moved HTTP and Stream upstream to separated include config file
Added DNS resolver support for stream servers and stream upstreams
Moved .declaration.http.resolvers[] to .declaration.resolvers[]
Postman collection updated for v5.1

* 2024080801 - 5.1 dev
Postman collection updated
NGINX One REST API endpoints updated

* 2024080809 - 5.1 dev
FEATURES updated
Postman collection README updated
API v5.1 USAGE updated

* 2024080902 - 5.1 dev
FEATURES updated

* 2024080903 - 5.1 dev

Author: fabriziofiorucci

.gitignore

@@ -24,4 +24,5 @@ Thumbs.db
 /src/v5_0/__pycache__/
 /src/v5_1/__pycache__/
 /contrib/devportal/redocly/src/__pycache__/
+/contrib/app-protect/src/__pycache__/
 /venv/

FEATURES.md

@@ -2,19 +2,18 @@
 
 [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
 
-This project provides a set of declarative REST API for [NGINX Instance Manager](https://docs.nginx.com/nginx-management-suite/nim/) and [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/) (currently in early stage).
+This project provides a set of declarative REST API for [NGINX Instance Manager](https://docs.nginx.com/nginx-management-suite/nim/) and [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/) (currently in early stage / unstable).
 
 It can be used to manage NGINX Plus configuration lifecycle and to create NGINX Plus configurations using JSON service definitions.
 
 GitOps integration is supported: source of truth is checked for updates (NGINX App Protect policies, TLS certificates, keys and chains/bundles, Swagger/OpenAPI definitions, snippets) and NGINX configurations are automatically kept in sync.
 
 Use cases include:
 
-- Rapid configuration generation and templating
-- CI/CD integration with NGINX Instance Manager (instance groups and staged configs) and NGINX One Cloud Console (clusters)
+- Integration with NGINX Instance Manager (instance group) and NGINX One Cloud Console (config sync group)
 - NGINX App Protect DevSecOps integration (NGINX Instance Manager only)
 - API Gateway deployments with automated Swagger / OpenAPI schema import
-- API Developer portals zero-touch deployment
+- API Developer portals zero-touch deployment (redocly and backstage supported)
 - GitOps integration with source of truth support for
   - NGINX App Protect WAF policies
   - TLS certificates, keys and chains/bundles
@@ -29,9 +28,9 @@ A **blog article** to automate NGINX API Gateway management from OpenAPI schemas
 ## Supported releases
 
 - NGINX Instance Manager 2.14+
-- NGINX One Cloud Console
+- NGINX One Cloud Console (early stage / unstable)
 - NGINX Plus R30+
-- NGINX App Protect WAF 4.8+
+- NGINX App Protect WAF 4
 
 ## Architecture
 
@@ -55,10 +54,6 @@ stateDiagram-v2
     DEVP: Developer Portal Service
     OUTPUT: Output
     REDIS: Redis
-    POST: Generic POST endpoint
-    CONFIGMAP: Kubernetes ConfigMap
-    PLAINTEXT: Plaintext
-    BASE64: Base64-encoded
 
     DevOps --> Pipeline
     Pipeline --> INPUT
@@ -69,10 +64,6 @@ stateDiagram-v2
     SOT --> NDAPI
     NDAPI --> REDIS
     REDIS --> NDAPI
-    OUTPUT --> BASE64
-    OUTPUT --> POST
-    OUTPUT --> CONFIGMAP
-    OUTPUT --> PLAINTEXT
     OUTPUT --> NIM
     OUTPUT --> N1
     NDAPI --> DEVP
@@ -124,7 +115,7 @@ critical If Developer Portal enabled
 end
 end
 
-NGINX Declarative API Core ->>+ NGINX Instance Manager / NGINX One: Publish staged config to instance group / cluster
+NGINX Declarative API Core ->>+ NGINX Instance Manager / NGINX One: Publish staged config to instance group / config sync group
 NGINX Instance Manager / NGINX One ->> NGINX: Publish config to NGINX instances
 NGINX Instance Manager / NGINX One ->>- NGINX Declarative API Core: Publish outcome
 
@@ -139,12 +130,8 @@ end
 
 ## Output formats
 
-- [X] Plaintext
-- [X] JSON-wrapped Base64-encoded
-- [X] Kubernetes Configmap
-- [X] POST to Generic REST API endpoint
-- [X] Output to NGINX Instance Manager 2.14+ imperative REST API (NGINX instance groups)
-- [X] Output to NGINX One Cloud Console REST API (NGINX clusters)
+- [X] Output to NGINX Instance Manager 2.14+ imperative REST API (instance group)
+- [X] Output to NGINX One Cloud Console REST API (config sync group)
 
 ## Supported features
 
@@ -154,9 +141,9 @@ See the [features list](/FEATURES.md)
 
 Usage details and JSON schema are available here:
 
-- [API v5.1](/USAGE-v5.1.md) - development
-- [API v5.0](/USAGE-v5.0.md) - latest
-- [API v4.2](/USAGE-v4.2.md)
+- [API v5.1](/USAGE-v5.1.md) - latest
+- [API v5.0](/USAGE-v5.0.md)
+- [API v4.2](/USAGE-v4.2.md) - deprecated
 
 A sample Postman collection and usage instructions can be found [here](/contrib/postman)
 

README.md

@@ -3,21 +3,13 @@
 Version 5.1 supports:
 
 - [NGINX Instance Manager](https://docs.nginx.com/nginx-management-suite/nim/) 2.14+
-- [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/)
+- [NGINX One Cloud Console](https://docs.nginx.com/nginx-one/) - early stage / unstable
 - [NGINX Plus](https://docs.nginx.com/nginx/) R30+
-- [NGINX App Protect WAF](https://docs.nginx.com/nginx-app-protect-waf/) 4.2.0+ with compiled [policy bundles](https://docs.nginx.com/nginx-app-protect-waf/v5/admin-guide/compiler/)
+- [NGINX App Protect WAF](https://docs.nginx.com/nginx-app-protect-waf/) 4 with precompiled [policy bundles](https://docs.nginx.com/nginx-app-protect-waf/v5/admin-guide/compiler/)
 
-The JSON schema is self explanatory. See also the [sample Postman collection](/contrib/postman)
+The JSON schema is self explanatory. See also the [sample Postman collection](/contrib/postman) for usage examples
 
 - `.output.type` defines how NGINX configuration will be returned:
-  - *plaintext* - plaintext format
-  - *json* - JSON-wrapped, base64-encoded
-  - *configmap* - Kubernetes Configmap in YAML format.
-    - `.output.configmap.name` must be set to the ConfigMap name
-    - `.output.configmap.filename` must be set to the NGINX configuration filename
-    - `.output.configmap.namespace` the optional namespace for the ConfigMap
-  - *http* - NGINX configuration is POSTed to custom url
-    - `.output.http.url` the URL to POST the configuration to
   - *nms* - NGINX configuration is published as a Staged Config to NGINX Instance Manager
     - `.output.nms.url` the NGINX Instance Manager URL
     - `.output.nms.username` the NGINX Instance Manager authentication username
@@ -38,165 +30,23 @@ The JSON schema is self explanatory. See also the [sample Postman collection](/c
       - `.output.nms.policies[].versions[].displayName` the policy version's display name
       - `.output.nms.policies[].versions[].description` the policy version's description
       - `.output.nms.policies[].versions[].contents` this can be either base64-encoded or be a HTTP(S) URL that will be fetched dynamically from a source of truth
-  - *nginxone* - NGINX configuration is published to a NGINX One Cloud Console cluster
+  - *nginxone* - NGINX configuration is published to a NGINX One Cloud Console config sync group
     - `.output.nginxone.url` the NGINX One Cloud Console URL
     - `.output.nginxone.namespace` the NGINX One Cloud Console namespace
     - `.output.nginxone.token` the authentication token
-    - `.output.nginxone.cluster` the cluster name
+    - `.output.nginxone.configsyncgroup` the NGINX One Cloud Console config sync group name
     - `.output.nginxone.synctime` **optional**, used for GitOps autosync. When specified and the declaration includes HTTP(S) references to NGINX App Protect policies, TLS certificates/keys/chains, the HTTP(S) endpoints will be checked every `synctime` seconds and if external contents have changed, the updated configuration will automatically be published to NGINX One Cloud Console
     - `.output.nginxone.modules` an optional array of NGINX module names (ie. 'ngx_http_app_protect_module', 'ngx_http_js_module','ngx_stream_js_module')
     - `.output.nginxone.certificates` an optional array of TLS certificates/keys/chains to be published
       - `.output.nginxone.certificates[].type` the item type ('certificate', 'key', 'chain')
       - `.output.nginxone.certificates[].name` the certificate/key/chain name with no path/extension (ie. 'test-application')
       - `.output.nginxone.certificates[].contents` the content: this can be either base64-encoded or be a HTTP(S) URL that will be fetched dynamically from a source of truth
-- `.declaration` describes the NGINX configuration to be created.
+- `.declaration` describes the NGINX configuration to be created
+  - `.declaration.http[]` NGINX HTTP definitions
+  - `.declaration.layer4[]` NGINX TCP/UDP definitions
+  - `.declaration.resolvers[]` DNS resolvers definitions
 
-### Locations ###
-
-Locations `.declaration.http.servers[].locations[].uri` match modifiers in `.declaration.http.servers[].locations[].urimatch` can be:
-
-- *prefix* - prefix URI matching
-- *exact* - exact URI matching
-- *regex* - case sensitive regex matching
-- *iregex* - case insensitive regex matching
-- *best* - case sensitive regex matching that halts any other location matching once a match is made
-
-### Javascript profiles ###
-
-NGINX Javascript profiles are defined in `.declaration.http.njs[]`:
-
-- `name` - the NJS profile name
-- `file.content` - the base64-encoded njs source code or the `http(s)://` URL of the file
-- `file.authentication.server[0].profile` - authentication profile name if `file.content` is a URL and the request must be authenticated
-
-### Javascript hooks ###
-
-NGINX Javascript hooks can be used in:
-
-- `.declaration.http.njs`
-  - Supported hooks:
-    - `js_preload_object'
-    - 'js_set`
-- `.declaration.http.server[].njs`
-  - Supported hooks:
-    - `js_preload_object'
-    - 'js_set`
-- `.declaration.http.server[].location[].njs`
-  - Supported hooks:
-    - `js_body_filter'
-    - 'js_content'
-    - 'js_header_filter'
-    - 'js_periodic'
-    - 'js_preload_object'
-    - 'js_set`
-
-Hooks invocation is:
-
-```
-"njs": [
-  {
-    "hook": {
-      "name": "<HOOK_NAME>",
-      "parameters": [
-        {
-          "name": "<HOOK_PARAMETER_NAME>",
-          "value": "<HOOK_PARAMETER_VALUE>"
-        }
-      ]
-    },
-    "profile": "<NGINX_JAVASCRIPT_PROFILE>",
-    "function": "<JAVASCRIPT_FUNCTION_NAME>"
-  }
-]
-```
-
-For detailed examples see the [Postman collection](/contrib/postman)
-
-### API Gateway ###
-
-Swagger files and OpenAPI schemas can be used to automatically configure NGINX as an API Gateway. Developer portal creation is supported through [Redocly](https://redocly.com/)
-
-Declaration path `.declaration.http.servers[].locations[].apigateway` defines the API Gateway configuration:
-
-- `openapi_schema` - the base64-encoded schema, or the schema URL. YAML and JSON are supported
-- `api_gateway.enabled` - enable/disable API Gateway provisioning
-- `api_gateway.strip_uri` - removes the `.declaration.http.servers[].locations[].uri` part of the URI before forwarding requests to the upstream
-- `api_gateway.server_url` - the base URL of the upstream server
-- `developer_portal.enabled` - enable/disable Developer portal provisioning
-- `developer_portal.type` - developer portal type. Currently supported are: `redocly`, `backstage`
-- `developer_portal.redocly.*` - Redocly-based developer portal parameters. See the [Postman collection](/contrib/postman)
-- `developer_portal.backstage.*` - Backstage-based developer portal parameters. See the [Postman collection](/contrib/postman)
-- `authentication` - optional, used to enforce authentication at the API Gateway level
-- `authentication.client[]` - authentication profile names
-- `authentication.enforceOnPaths` - if set to `true` authentication is enforced on all API endpoints listed under `authentication.paths`. if set to `false` authentication is enforced on all API endpoints but those listed under `authentication.paths`
-- `authentication.paths` - paths to enforce authentication
-- `authorization[]` - optional, used to enforce authorization
-- `authorization[].profile` - authorization profile name
-- `authorization[].enforceOnPaths` - if set to `true` authorization is enforced on all API endpoints listed under `authorization.paths`. if set to `false` authorization is enforced on all API endpoints but those listed under `authorization[].paths`
-- `authorization[].paths` - paths to enforce authorization
-- `rate_limit` - optional, used to enforce rate limiting at the API Gateway level
-- `rate_limit.enforceOnPaths` - if set to `true` rate limiting is enforced on all API endpoints listed under `rate_limit.paths`. if set to `false` rate limiting is enforced on all API endpoints but those listed under `rate_limit.paths`
-
-A sample API Gateway declaration to publish the `https://petstore.swagger.io` REST API using:
-
-- REST API endpoint URIs
-- HTTP Methods
-- Rate limiting on `/user/login`, `/usr/logout` and `/pet/{petId}/uploadImage`
-- JWT authentication on `/user/login`, `/usr/logout` and `/pet/{petId}/uploadImage`
-- JWT claim-based authorization on `/user/login`, `/usr/logout` and `/pet/{petId}/uploadImage`
-- Redocly-based developer portal
-- NGINX App Protect WAF security
-
-can be found in the [Postman collection](/contrib/)
-
-### Maps ###
-
-Map entries `.declaration.maps[].entries.keymatch` can be:
-
-- *exact* - exact variable matching
-- *regex* - case sensitive regex matching
-- *iregex* - case insensitive regex matching
-
-### Snippets ###
-
-Snippets for `http`, `upstream`, `server` and `location` can be specified as:
-- base64-encoded content
-- HTTP(S) URL of a source of truth to fetch snippet content from. Content on the source of truth must be plaintext
-- source of truth authentication is supported through authentication profiles
-
-### Authentication profiles ###
-
-Client and Server authentication profiles can be defined in the declarative json at `.declaration.http.authentication`
-
-```commandline
-  "authentication": {
-    "client": [
-        {
-            "name": "<PROFILE_NAME>",
-            "type": "<PROFILE_TYPE>",
-            "<PROFILE_TYPE>": {
-                "<PROFILE_KEY>": "<VALUE>",
-                [...]
-            }
-        },
-        [...]
-    ],
-    "server": [
-        {
-            "name": "<PROFILE_NAME>",
-            "type": "<PROFILE_TYPE>",
-            "<PROFILE_TYPE>": {
-                "<PROFILE_KEY>": "<VALUE>",
-                [...]
-            }
-        },
-        [...]
-    ]
-```
-
-For a list of all supported authentication profile types see the [feature matrix](/FEATURES.md)
-
-### API endpoints ###
+### API endpoints
 
 - `POST /v5.1/config/` - Publish a new declaration
 - `PATCH /v5.1/config/{config_uid}` - Update an existing declaration
@@ -206,8 +56,4 @@ For a list of all supported authentication profile types see the [feature matrix
   - Per-Stream upstream CRUD
   - Per-NGINX App Protect WAF policy CRUD
 - `GET /v5.1/config/{config_uid}` - Retrieve an existing declaration
-- `DELETE /v5.1/config/{config_uid}` - Delete an existing declaration
-
-### Usage Examples ###
-
-A sample Postman collection is available [here](/contrib/postman)
+- `DELETE /v5.1/config/{config_uid}` - Delete an existing declaration

USAGE-v5.1.md

@@ -0,0 +1,53 @@
+# syntax=docker/dockerfile:1
+ARG VERSION_TAG
+ARG BASE_IMAGE=private-registry.nginx.com/nap/waf-compiler:${VERSION_TAG}
+FROM ${BASE_IMAGE}
+
+# Installing packages as root
+USER root
+
+ENV DEBIAN_FRONTEND="noninteractive"
+
+# REST API wrapper
+WORKDIR /compiler
+COPY src src/
+# REST API wrapper
+
+RUN --mount=type=secret,id=nginx-crt,dst=/etc/ssl/nginx/nginx-repo.crt,mode=0644 \
+    --mount=type=secret,id=nginx-key,dst=/etc/ssl/nginx/nginx-repo.key,mode=0644 \
+    apt-get update \
+    && apt-get install -y \
+        apt-transport-https \
+        lsb-release \
+        ca-certificates \
+        wget \
+        gnupg2 \
+        ubuntu-keyring \
+    && wget -qO - https://cs.nginx.com/static/keys/app-protect-security-updates.key | gpg --dearmor | \
+    tee /usr/share/keyrings/app-protect-security-updates.gpg >/dev/null \
+    && printf "deb [signed-by=/usr/share/keyrings/app-protect-security-updates.gpg] \
+    https://pkgs.nginx.com/app-protect-security-updates/ubuntu `lsb_release -cs` nginx-plus\n" | \
+    tee /etc/apt/sources.list.d/nginx-app-protect.list \
+    && wget -P /etc/apt/apt.conf.d https://cs.nginx.com/static/files/90pkgs-nginx \
+    && apt-get update \
+    && apt-get install -y \
+        app-protect-attack-signatures \
+        app-protect-bot-signatures \
+        app-protect-threat-campaigns \
+# REST API wrapper
+    && apt-get -y install python3 python3-venv \
+    && python3 -m venv /compiler/env/ \
+    && . /compiler/env/bin/activate \
+    && pip3 install --no-cache --upgrade pip setuptools virtualenv \
+    && python3 -m pip install --upgrade pip \
+    && pip3 install -r /compiler/src/requirements.txt \
+# REST API wrapper
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# non-root default user (UID 101)
+USER nginx
+
+# REST API wrapper
+ENTRYPOINT [ "/compiler/src/start.sh" ]
+# REST API wrapper

contrib/app-protect/Dockerfile

@@ -0,0 +1,21 @@
+# REST API for NGINX App Protect WAF Compiler
+
+This contrib provides a set of REST API to use the NGINX App Protect 5 policy compiler
+
+## REST API Endpoints
+
+- `/v1/compile/policy` - compiles a JSON policy into a bundle
+  - Method: `POST`
+  - Payload: `{"global-settings": "<BASE64_ENCODED_GLOBAL_SETTINGS_JSON>", "policy": "<BASE64_ENCODED_POLICY_JSON>", "cookie-protection-seed": "<SEED_VALUE>"}`
+- `/v1/compile/logprofile` - compiles a log profile JSON into a bundle
+  - Method: `POST`
+  - Payload: `{"logprofile": "<BASE64_ENCODED_LOG_PROFILE_JSON>"}`
+- `/v1/bundle/info` - returns details on a compiled bundle
+  - Method: `POST`
+  - Payload: `{"bundle": "<BASE64_ENCODED_TGZ_BUNDLE>"}`
+
+Headers required for all endpoints:
+
+```
+Content-Type: application/json
+```

contrib/app-protect/README.md

@@ -0,0 +1,5 @@
+typing
+uvicorn
+fastapi
+uuid
+pyyaml