Materialize on Google Cloud Platform

Terraform module for deploying Materialize on Google Cloud Platform (GCP) with all required infrastructure components.

This module sets up:

• GKE cluster for Materialize workloads
• Cloud SQL PostgreSQL instance for metadata storage
• Cloud Storage bucket for persistence
• Required networking and security configurations
• Service accounts with proper IAM permissions

Warning

This module is intended for demonstration/evaluation purposes as well as for serving as a template when building your own production deployment of Materialize.

This module should not be directly relied upon for production deployments: future releases of the module will contain breaking changes. Instead, to use as a starting point for your own production deployment, either:

• Fork this repo and pin to a specific version, or
• Use the code as a reference when developing your own deployment.

The module has been tested with:

• GKE version 1.28
• PostgreSQL 15
• terraform-helm-materialize v0.1.12 (Materialize Operator v25.1.7)

Disk Support for Materialize on GCP

This module supports configuring disk support for Materialize using local SSDs in GCP with OpenEBS and lgalloc.

Machine Types with Local SSDs in GCP

When using disk support for Materialize on GCP, you need to use machine types that support local SSD attachment. Here are some recommended machine types:

• N2 series with local NVMe SSDs:

  • For memory-optimized workloads, consider n2-highmem-16 or n2-highmem-32 with local NVMe SSDs
  • Example: n2-highmem-32 with 2 or more local SSDs

• N2D series with local NVMe SSDs:

  • For memory-optimized workloads, consider n2d-highmem-16 or n2d-highmem-32 with local NVMe SSDs
  • Example: n2d-highmem-32 with 2 or more local SSDs

Enabling Disk Support

To enable disk support with default settings in your Terraform configuration:

enable_disk_support = true

gke_config = {
  node_count   = 3

  # This machine has 256GB RAM
  machine_type = "n2-highmem-32"

  # This is for the OS disk, not for Materialize data
  disk_size_gb = 100
  min_nodes    = 3
  max_nodes    = 5

  # This provides 2 x 375GB = 750GB of local SSD storage
  # Exceeding the 2:1 disk-to-RAM ratio (256GB RAM : 750GB disk)
  local_ssd_count = 2
}

This configuration:

1. Attaches two local SSDs to each node, providing 750GB of storage per node
2. Ensures the disk-to-RAM ratio is greater than 2:1 for the n2-highmem-32 instance (which has 256GB RAM)
3. Installs OpenEBS via Helm to manage these local SSDs
4. Configures local NVMe SSD devices using the bootstrap script
5. Creates appropriate storage classes for Materialize

Advanced Configuration Example

For a different machine type with appropriate disk sizing:

enable_disk_support = true

gke_config = {
  node_count   = 3
  # This machine has 128GB RAM
  machine_type = "n2-highmem-16"
  disk_size_gb = 100
  min_nodes    = 3
  max_nodes    = 5
  # This provides 1 x 375GB = 375GB of local NVMe SSD storage
  # Exceeding the 2:1 disk-to-RAM ratio (128GB RAM : 375GB disk)
  local_ssd_count = 1
}

disk_support_config = {
  openebs_version    = "4.2.0"
  storage_class_name = "custom-storage-class"
}

Calculating the Right Number of Local SSDs

The following table helps you determine the appropriate number of local SSDs based on your chosen machine type to maintain the recommended 2:1 disk-to-RAM ratio:

Machine Type	RAM	Required Disk	Recommended Local SSD Count	Total SSD Storage
n2-highmem-8	64GB	128GB	1	375GB
n2-highmem-16	128GB	256GB	1	375GB
n2-highmem-32	256GB	512GB	2	750GB
n2-highmem-64	512GB	1024GB	3	1125GB
n2-highmem-80	640GB	1280GB	4	1500GB

Remember that each local NVMe SSD in GCP provides 375GB of storage. Choose the appropriate local_ssd_count to make sure your total disk space is at least twice the amount of RAM in your machine type for optimal Materialize performance.

Requirements

Name	Version
terraform	>= 1.0
google	>= 6.0
helm	~> 2.0
kubernetes	~> 2.0

Providers

No providers.

Modules

Name	Source	Version
certificates	./modules/certificates	n/a
database	./modules/database	n/a
gke	./modules/gke	n/a
load_balancers	./modules/load_balancers	n/a
networking	./modules/networking	n/a
operator	github.com/MaterializeInc/terraform-helm-materialize	v0.1.12
storage	./modules/storage	n/a

Resources

No resources.

Inputs

Name	Description	Type	Default	Required
cert_manager_chart_version	Version of the cert-manager helm chart to install.	string	"v1.17.1"	no
cert_manager_install_timeout	Timeout for installing the cert-manager helm chart, in seconds.	number	300	no
cert_manager_namespace	The name of the namespace in which cert-manager is or will be installed.	string	"cert-manager"	no
database_config	Cloud SQL configuration	object({ tier = optional(string, "db-custom-2-4096") version = optional(string, "POSTGRES_15") password = string username = optional(string, "materialize") db_name = optional(string, "materialize") })	n/a	yes
disk_support_config	Advanced configuration for disk support (only used when enable_disk_support = true)	object({ install_openebs = optional(bool, true) run_disk_setup_script = optional(bool, true) local_ssd_count = optional(number, 1) create_storage_class = optional(bool, true) openebs_version = optional(string, "4.2.0") openebs_namespace = optional(string, "openebs") storage_class_name = optional(string, "openebs-lvm-instance-store-ext4") })	{}	no
enable_disk_support	Enable disk support for Materialize using OpenEBS and local SSDs. When enabled, this configures OpenEBS, runs the disk setup script, and creates appropriate storage classes.	bool	true	no
gke_config	GKE cluster configuration. Make sure to use large enough machine types for your Materialize instances.	object({ node_count = number machine_type = string disk_size_gb = number min_nodes = number max_nodes = number })	{ "disk_size_gb": 100, "machine_type": "n2-highmem-8", "max_nodes": 2, "min_nodes": 1, "node_count": 1 }	no
helm_chart	Chart name from repository or local path to chart. For local charts, set the path to the chart directory.	string	"materialize-operator"	no
helm_values	Values to pass to the Helm chart	any	{}	no
install_cert_manager	Whether to install cert-manager.	bool	true	no
install_materialize_operator	Whether to install the Materialize operator	bool	true	no
install_metrics_server	Whether to install the metrics-server for the Materialize Console. Defaults to false since GKE installs one by default in the kube-system namespace. Only set to true if the GKE cluster was deployed with monitoring explicitly turned off. Refer to the GKE docs for more information, including impact to GKE customer support efforts.	bool	false	no
labels	Labels to apply to all resources	map(string)	{}	no
materialize_instances	Configuration for Materialize instances	list(object({ name = string namespace = optional(string) database_name = string create_database = optional(bool, true) create_load_balancer = optional(bool, true) internal_load_balancer = optional(bool, true) environmentd_version = optional(string) cpu_request = optional(string, "1") memory_request = optional(string, "1Gi") memory_limit = optional(string, "1Gi") in_place_rollout = optional(bool, false) request_rollout = optional(string) force_rollout = optional(string) balancer_memory_request = optional(string, "256Mi") balancer_memory_limit = optional(string, "256Mi") balancer_cpu_request = optional(string, "100m") license_key = optional(string) }))	[]	no
namespace	Kubernetes namespace for Materialize	string	"materialize"	no
network_config	Network configuration for the GKE cluster	object({ subnet_cidr = string pods_cidr = string services_cidr = string })	n/a	yes
operator_namespace	Namespace for the Materialize operator	string	"materialize"	no
operator_version	Version of the Materialize operator to install	string	null	no
orchestratord_version	Version of the Materialize orchestrator to install	string	null	no
prefix	Prefix to be used for resource names	string	"materialize"	no
project_id	The ID of the project where resources will be created	string	n/a	yes
region	The region where resources will be created	string	"us-central1"	no
storage_bucket_version_ttl	Sets the TTL (in days) on non current storage bucket objects. This must be set if storage_bucket_versioning is turned on.	number	7	no
storage_bucket_versioning	Enable bucket versioning. This should be enabled for production deployments.	bool	false	no
use_local_chart	Whether to use a local chart instead of one from a repository	bool	false	no
use_self_signed_cluster_issuer	Whether to install and use a self-signed ClusterIssuer for TLS. To work around limitations in Terraform, this will be treated as false if no materialize instances are defined.	bool	true	no

Outputs

Name	Description
connection_strings	Formatted connection strings for Materialize
database	Cloud SQL instance details
gke_cluster	GKE cluster details
load_balancer_details	Details of the Materialize instance load balancers.
network	Network details
operator	Materialize operator details
service_accounts	Service account details
storage	GCS bucket details

Connecting to Materialize instances

Access to the database is through the balancerd pods on:

• Port 6875 for SQL connections.
• Port 6876 for HTTP(S) connections.

Access to the web console is through the console pods on port 8080.

TLS support

TLS support is provided by using cert-manager and a self-signed ClusterIssuer.

More advanced TLS support using user-provided CAs or per-Materialize Issuers are out of scope for this Terraform module. Please refer to the cert-manager documentation for detailed guidance on more advanced usage.

Upgrade Notes

v0.3.0

We now install cert-manager and configure a self-signed ClusterIssuer by default.

Due to limitations in Terraform, it cannot plan Kubernetes resources using CRDs that do not exist yet. We have worked around this for new users by only generating the certificate resources when creating Materialize instances that use them, which also cannot be created on the first run.

For existing users upgrading Materialize instances not previously configured for TLS:

1. Leave install_cert_manager at its default of true.
2. Set use_self_signed_cluster_issuer to false.
3. Run terraform apply. This will install cert-manager and its CRDs.
4. Set use_self_signed_cluster_issuer back to true (the default).
5. Update the request_rollout field of the Materialize instance.
6. Run terraform apply. This will generate the certificates and configure your Materialize instance to use them.

Storage Bucket Versioning

By default storage bucket versioning is turned off. This both reduces costs and allows for easier cleanup of resources for testing. When running in production, versioning should be turned on with a sufficient TTL to meet any data-recovery requirements. See storage_bucket_versioning and storage_bucket_version_ttl.