Implement provisioner (#683)

Problem:
Support provisioning NGINX data plane per Gateway, as expected
by the Gateway API conformance tests.
See #634
for more info

Solution:
- Implement provisioner command which which provisions a Deployment
of NKG (static mode) for each Gateway of the provisioner GatewayClass.
- Add provisioner manifests and docs

Fixes #634

Additionally, introduce PrepareTimeForFakeClient helper function, which
fixes an error that appeared on GitHub Action pipeline, not locally (see below).
(To reproduce it locally, run `make TZ=123 unit-tests` from a commit before
this one and ensure you compare Conditions in the status as in
Expect(clusterGc.Status.Conditions).To(Equal(expectedConditions)) )

The timezone of the time in a resource field returned by the fake client was different
from the one set in the field when updating the resource.

The commit adds PrepareTimeForFakeClient() which ensures that the time is prepared
correctly so that the timezone is the same.

The problem is only present when comparing status Conditions  using gomega like
Expect(clusterGc.Status.Conditions).To(Equal(expectedConditions))
but not present if comparing using cmp like
Expect(helpers.Diff(expectedGc, latestGc)).To(BeEmpty()).

  [FAILED] Expected
      <*time.Location | 0x30d0b00>: {
          name: "Local",
          zone: [
              {name: "UTC", offset: 0, isDST: false},
          ],
          tx: [
              {
                  when: -9223372036854775808,
                  index: 0,
                  isstd: false,
                  isutc: false,
              },
          ],
          extend: "UTC0",
          cacheStart: -9223372036854775808,
          cacheEnd: 9223372036854775807,
          cacheZone: {name: "UTC", offset: 0, isDST: false},
      }
  to equal
      <*time.Location | 0x309f240>: {name: "UTC", zone: nil, tx: nil, extend: "", cacheStart: 0, cacheEnd: 0, cacheZone: nil}

Author: pleshakov

cmd/gateway/commands.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"errors"
 	"fmt"
 	"os"
 
@@ -12,6 +11,7 @@ import (
 
 	"github.com/nginxinc/nginx-kubernetes-gateway/internal/config"
 	"github.com/nginxinc/nginx-kubernetes-gateway/internal/manager"
+	"github.com/nginxinc/nginx-kubernetes-gateway/internal/provisioner"
 )
 
 const (
@@ -190,7 +190,10 @@ func createProvisionerModeCommand() *cobra.Command {
 				"date", date,
 			)
 
-			return errors.New("not implemented yet")
+			return provisioner.StartManager(provisioner.Config{
+				Logger:           logger,
+				GatewayClassName: gatewayClassName.value,
+			})
 		},
 	}
 }

conformance/provisioner/provisioner.md

@@ -0,0 +1,35 @@
+# Provisioner
+
+Provisioner implements data plane provisioning for NGINX Kubernetes Gateway (NKG): it creates an NKG static mode
+Deployment for each Gateway that belongs to the provisioner GatewayClass.
+
+```
+Usage:
+  gateway provisioner-mode [flags]
+
+Flags:
+  -h, --help   help for provisioner-mode
+
+Global Flags:
+      --gateway-ctlr-name string   The name of the Gateway controller. The controller name must be of the form: DOMAIN/PATH. The controller's domain is 'k8s-gateway.nginx.org' (default "")
+      --gatewayclass string        The name of the GatewayClass resource. Every NGINX Gateway must have a unique corresponding GatewayClass resource. (default "")
+```
+
+Provisioner is not meant to be used in production yet (see this issue for more details
+https://github.com/nginxinc/nginx-kubernetes-gateway/issues/634). However, it can be used in the Gateway API conformance
+tests, which expect a Gateway API implementation to provision an independent data plane per Gateway.
+
+How to deploy:
+
+1. Follow the [installation](/docs/installation.md) instructions up until the Deploy the NGINX Kubernetes Gateway Step
+   to deploy prerequisites for both the static mode Deployments and the provisioner.
+1. Deploy provisioner:
+   ```
+   kubectl apply -f conformance/provisioner/provisioner.yaml
+   ```
+1. Confirm the provisioner is running in nginx-gateway namespace:
+   ```
+   kubectl get pods -n nginx-gateway 
+   NAME                                         READY   STATUS    RESTARTS   AGE
+   nginx-gateway-provisioner-6c9d9fdcb8-b2pf8   1/1     Running   0          11m
+   ```

conformance/provisioner/provisioner.yaml

@@ -0,0 +1,72 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: nginx-gateway-provisioner
+  namespace: nginx-gateway
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: nginx-gateway-provisioner
+rules:
+- apiGroups:
+  - apps
+  resources:
+  - deployments
+  verbs:
+  - create
+  - delete
+- apiGroups:
+  - gateway.networking.k8s.io
+  resources:
+  - gatewayclasses
+  - gateways
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - gateway.networking.k8s.io
+  resources:
+  - gatewayclasses/status
+  verbs:
+  - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: nginx-gateway-provisioner
+subjects:
+- kind: ServiceAccount
+  name: nginx-gateway-provisioner
+  namespace: nginx-gateway
+roleRef:
+  kind: ClusterRole
+  name: nginx-gateway-provisioner
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nginx-gateway-provisioner
+  namespace: nginx-gateway
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: nginx-gateway-provisioner
+  template:
+    metadata:
+      labels:
+        app: nginx-gateway-provisioner
+    spec:
+      serviceAccountName: nginx-gateway-provisioner
+      containers:
+      - image: ghcr.io/nginxinc/nginx-kubernetes-gateway:edge
+        imagePullPolicy: Always
+        name: nginx-gateway-provisioner
+        securityContext:
+          runAsUser: 1001
+        args:
+        - provisioner-mode
+        - --gateway-ctlr-name=k8s-gateway.nginx.org/nginx-gateway-controller
+        - --gatewayclass=nginx

deploy/manifests/nginx-gateway.yaml deploy/manifests/deployment.yaml

@@ -1,95 +1,3 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: nginx-gateway
-  namespace: nginx-gateway
----
-kind: ClusterRole
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: nginx-gateway
-rules:
-- apiGroups:
-  - ""
-  resources:
-  - services
-  - secrets
-  verbs:
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - events
-  verbs:
-  - create
-  - patch
-- apiGroups:
-  - discovery.k8s.io
-  resources:
-  - endpointslices
-  verbs:
-  - list
-  - watch
-- apiGroups:
-  - gateway.networking.k8s.io
-  resources:
-  - gatewayclasses
-  - gateways
-  - httproutes
-  verbs:
-  - list
-  - watch
-- apiGroups:
-  - gateway.nginx.org
-  resources:
-  - gatewayconfigs
-  verbs:
-  - list
-  - watch
-- apiGroups:
-  - gateway.networking.k8s.io
-  resources:
-  - httproutes/status
-  - gateways/status
-  - gatewayclasses/status
-  verbs:
-  - update
----
-kind: ClusterRoleBinding
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: nginx-gateway
-subjects:
-- kind: ServiceAccount
-  name: nginx-gateway
-  namespace: nginx-gateway
-roleRef:
-  kind: ClusterRole
-  name: nginx-gateway
-  apiGroup: rbac.authorization.k8s.io
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: nginx-conf
-  namespace: nginx-gateway
-data:
-  nginx.conf: |
-    load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
-
-    events {}
-
-    pid /etc/nginx/nginx.pid;
-    error_log stderr debug;
-
-    http {
-      include /etc/nginx/conf.d/*.conf;
-      js_import /usr/lib/nginx/modules/njs/httpmatches.js;
-      server_names_hash_bucket_size 256;
-      server_names_hash_max_size 1024;
-    }
----
 apiVersion: apps/v1
 kind: Deployment
 metadata:

deploy/manifests/nginx-conf.yaml

@@ -0,0 +1,20 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nginx-conf
+  namespace: nginx-gateway
+data:
+  nginx.conf: |
+    load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
+
+    events {}
+
+    pid /etc/nginx/nginx.pid;
+    error_log stderr debug;
+
+    http {
+      include /etc/nginx/conf.d/*.conf;
+      js_import /usr/lib/nginx/modules/njs/httpmatches.js;
+      server_names_hash_bucket_size 256;
+      server_names_hash_max_size 1024;
+    }

deploy/manifests/rbac.yaml

@@ -0,0 +1,70 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: nginx-gateway
+  namespace: nginx-gateway
+---
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: nginx-gateway
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - services
+  - secrets
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - events
+  verbs:
+  - create
+  - patch
+- apiGroups:
+  - discovery.k8s.io
+  resources:
+  - endpointslices
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - gateway.networking.k8s.io
+  resources:
+  - gatewayclasses
+  - gateways
+  - httproutes
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - gateway.nginx.org
+  resources:
+  - gatewayconfigs
+  verbs:
+  - list
+  - watch
+- apiGroups:
+  - gateway.networking.k8s.io
+  resources:
+  - httproutes/status
+  - gateways/status
+  - gatewayclasses/status
+  verbs:
+  - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: nginx-gateway
+subjects:
+- kind: ServiceAccount
+  name: nginx-gateway
+  namespace: nginx-gateway
+roleRef:
+  kind: ClusterRole
+  name: nginx-gateway
+  apiGroup: rbac.authorization.k8s.io

docs/installation.md

@@ -35,6 +35,18 @@ This guide walks you through how to install NGINX Kubernetes Gateway on a generi
     kubectl create configmap njs-modules --from-file=internal/nginx/modules/src/httpmatches.js -n nginx-gateway
     ```
 
+1. Create the ConfigMap with the main NGINX configuration file:
+
+    ```
+    kubectl apply -f deploy/manifests/nginx-conf.yaml
+    ```
+
+1. Configure RBAC:
+
+    ```
+    kubectl apply -f deploy/manifests/rbac.yaml
+    ```
+
 1. Create the GatewayClass resource:
 
     ```
@@ -44,7 +56,7 @@ This guide walks you through how to install NGINX Kubernetes Gateway on a generi
 1. Deploy the NGINX Kubernetes Gateway:
 
    ```
-   kubectl apply -f deploy/manifests/nginx-gateway.yaml
+   kubectl apply -f deploy/manifests/deployment.yaml
    ```
 
 1. Confirm the NGINX Kubernetes Gateway is running in `nginx-gateway` namespace:

embedded.go

@@ -0,0 +1,11 @@
+package embeddedfiles
+
+import _ "embed"
+
+// StaticModeDeploymentYAML contains the YAML manifest of the Deployment resource for the static mode.
+//
+// We put this in the root of the repo because goembed doesn't support relative/absolute paths and symlinks,
+// and we want to keep the manifests in the deploy/manifests directory.
+//
+//go:embed deploy/manifests/deployment.yaml
+var StaticModeDeploymentYAML []byte

internal/helpers/helpers.go

@@ -2,7 +2,10 @@
 package helpers
 
 import (
+	"fmt"
+
 	"github.com/google/go-cmp/cmp"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"sigs.k8s.io/gateway-api/apis/v1beta1"
 )
 
@@ -62,3 +65,21 @@ func GetBoolPointer(b bool) *bool {
 func GetPointer[T any](v T) *T {
 	return &v
 }
+
+// PrepareTimeForFakeClient processes the time similarly to the fake client
+// from sigs.k8s.io/controller-runtime/pkg/client/fake
+// making it is possible to use it in tests when comparing against values returned by the fake client.
+// It panics if it fails to process the time.
+func PrepareTimeForFakeClient(t metav1.Time) metav1.Time {
+	bytes, err := t.Marshal()
+	if err != nil {
+		panic(fmt.Errorf("failed to marshal time: %w", err))
+	}
+
+	err = t.Unmarshal(bytes)
+	if err != nil {
+		panic(fmt.Errorf("failed to unmarshal time: %w", err))
+	}
+
+	return t
+}