feat: Add option to let the module manage the webhook secret

README.md

@@ -86,10 +86,12 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
 |------|--------|---------|
 | <a name="module_ami_housekeeper"></a> [ami\_housekeeper](#module\_ami\_housekeeper) | ./modules/ami-housekeeper | n/a |
 | <a name="module_instance_termination_watcher"></a> [instance\_termination\_watcher](#module\_instance\_termination\_watcher) | ./modules/termination-watcher | n/a |
+| <a name="module_rotating_random"></a> [rotating\_random](#module\_rotating\_random) | ./modules/rotating-random | n/a |
 | <a name="module_runner_binaries"></a> [runner\_binaries](#module\_runner\_binaries) | ./modules/runner-binaries-syncer | n/a |
 | <a name="module_runners"></a> [runners](#module\_runners) | ./modules/runners | n/a |
 | <a name="module_ssm"></a> [ssm](#module\_ssm) | ./modules/ssm | n/a |
 | <a name="module_webhook"></a> [webhook](#module\_webhook) | ./modules/webhook | n/a |
+| <a name="module_webhook_github_app"></a> [webhook\_github\_app](#module\_webhook\_github\_app) | ./modules/webhook-github-app | n/a |
 
 ## Resources
 
@@ -140,8 +142,8 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
 | <a name="input_enable_userdata"></a> [enable\_userdata](#input\_enable\_userdata) | Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI. | `bool` | `true` | no |
 | <a name="input_eventbridge"></a> [eventbridge](#input\_eventbridge) | Enable the use of EventBridge by the module. By enabling this feature events will be put on the EventBridge by the webhook instead of directly dispatching to queues for scaling.<br/><br/>    `enable`: Enable the EventBridge feature.<br/>    `accept_events`: List can be used to only allow specific events to be putted on the EventBridge. By default all events, empty list will be be interpreted as all events. | <pre>object({<br/>    enable        = optional(bool, true)<br/>    accept_events = optional(list(string), null)<br/>  })</pre> | `{}` | no |
 | <a name="input_ghes_ssl_verify"></a> [ghes\_ssl\_verify](#input\_ghes\_ssl\_verify) | GitHub Enterprise SSL verification. Set to 'false' when custom certificate (chains) is used for GitHub Enterprise Server (insecure). | `bool` | `true` | no |
-| <a name="input_ghes_url"></a> [ghes\_url](#input\_ghes\_url) | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB - github.com. However if you are using Github Enterprise Cloud with data-residency (ghe.com), set the endpoint here. Example - https://companyname.ghe.com | `string` | `null` | no |
-| <a name="input_github_app"></a> [github\_app](#input\_github\_app) | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`). | <pre>object({<br/>    key_base64     = string<br/>    id             = string<br/>    webhook_secret = string<br/>  })</pre> | n/a | yes |
+| <a name="input_ghes_url"></a> [ghes\_url](#input\_ghes\_url) | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB. However if you are using Github Enterprise Cloud with data-residency (ghe.com), set the endpoint here. Example - https://companyname.ghe.com | `string` | `null` | no |
+| <a name="input_github_app"></a> [github\_app](#input\_github\_app) | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`)."<br/><br/>    If `webhook_secret` is not set, a random secret will be generated and stored in SSM. The secret is used to validate the webhook events. If you want to use your own secret, set the `webhook_secret` parameter.<br/>    When the secret is managed by the module, it will be rotated every `webhook_secret_rotation_days` days. | <pre>object({<br/>    key_base64                   = string<br/>    id                           = string<br/>    webhook_secret               = optional(string)<br/>    webhook_secret_rotation_days = optional(number, 30)<br/>  })</pre> | n/a | yes |
 | <a name="input_idle_config"></a> [idle\_config](#input\_idle\_config) | List of time periods, defined as a cron expression, to keep a minimum amount of runners active instead of scaling down to 0. By defining this list you can ensure that in time periods that match the cron expression within 5 seconds a runner is kept idle. | <pre>list(object({<br/>    cron             = string<br/>    timeZone         = string<br/>    idleCount        = number<br/>    evictionStrategy = optional(string, "oldest_first")<br/>  }))</pre> | `[]` | no |
 | <a name="input_instance_allocation_strategy"></a> [instance\_allocation\_strategy](#input\_instance\_allocation\_strategy) | The allocation strategy for spot instances. AWS recommends using `price-capacity-optimized` however the AWS default is `lowest-price`. | `string` | `"lowest-price"` | no |
 | <a name="input_instance_max_spot_price"></a> [instance\_max\_spot\_price](#input\_instance\_max\_spot\_price) | Max price price for spot instances per hour. This variable will be passed to the create fleet as max spot price for the fleet. | `string` | `null` | no |

docs/configuration.md

@@ -14,6 +14,7 @@ To be able to support a number of use-cases, the module has quite a lot of confi
 - Spot vs on-demand. The runners use either the EC2 spot or on-demand life cycle. Runners will be created via the AWS [CreateFleet API](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html). The module (scale up lambda) will request via the CreateFleet API to create instances in one of the subnets and of the specified instance types.
 - ARM64 support via Graviton/Graviton2 instance-types. When using the default example or top-level module, specifying `instance_types` that match a Graviton/Graviton 2 (ARM64) architecture (e.g. a1, t4g or any 6th-gen `g` or `gd` type), you must also specify `runner_architecture = "arm64"` and the sub-modules will be automatically configured to provision with ARM64 AMIs and leverage GitHub's ARM64 action runner. See below for more details.
 - Disable default labels for the runners (os, architecture and `self-hosted`) can achieve by setting `runner_disable_default_labels` = true. If enabled, the runner will only have the extra labels provided in `runner_extra_labels`. In case you on own start script is used, this configuration parameter needs to be parsed via SSM.
+- Managed vs self-managed webhook secret. The module can manage the webhook secret for you. In that case simply do not provide a value for `github_app.webhook_secret`. If you want to manage the secret yourself, provide a value for `github_app.webhook_secret`. The secret will be managed and a rotation is triggered once running terraform apply again after `github_app.webhook_secret_rotation_days` days. **Important note**: THe managed webhook secret depends on a local-exec (bash) to update the secret in GitNub. It will also update the webhook url.
 
 ## AWS SSM Parameters
 

docs/getting-started.md

@@ -85,7 +85,7 @@ module "github-runner" {
   github_app = {
     key_base64     = "base64string"
     id             = "1"
-    webhook_secret = "webhook_secret"
+    webhook_secret = "webhook_secret" # optional, if not set the module will manage the secret.
   }
 
   webhook_lambda_zip                = "lambdas-download/webhook.zip"
@@ -109,7 +109,7 @@ The lambda for syncing the GitHub distribution to S3 is triggered via CloudWatch
 ### Setup the webhook / GitHub App (part 2)
 
 At this point you have two options. Either create a separate webhook (enterprise,
-org, or repo), or create a webhook in the App.
+org, or repo), or create a webhook in the App. In case you have not provided a Webhook secret the module will create one and update the GitHub app with both the secret and the webhook url.
 
 #### Option 1: Webhook
 

examples/default/README.md

@@ -40,23 +40,18 @@ terraform output -raw webhook_secret
 
 ## Providers
 
-| Name | Version |
-|------|---------|
-| <a name="provider_random"></a> [random](#provider\_random) | 3.6.3 |
+No providers.
 
 ## Modules
 
 | Name | Source | Version |
 |------|--------|---------|
 | <a name="module_base"></a> [base](#module\_base) | ../base | n/a |
 | <a name="module_runners"></a> [runners](#module\_runners) | ../../ | n/a |
-| <a name="module_webhook_github_app"></a> [webhook\_github\_app](#module\_webhook\_github\_app) | ../../modules/webhook-github-app | n/a |
 
 ## Resources
 
-| Name | Type |
-|------|------|
-| [random_id.random](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
+No resources.
 
 ## Inputs
 
@@ -72,5 +67,4 @@ terraform output -raw webhook_secret
 |------|-------------|
 | <a name="output_runners"></a> [runners](#output\_runners) | n/a |
 | <a name="output_webhook_endpoint"></a> [webhook\_endpoint](#output\_webhook\_endpoint) | n/a |
-| <a name="output_webhook_secret"></a> [webhook\_secret](#output\_webhook\_secret) | n/a |
 <!-- END_TF_DOCS -->

examples/default/main.tf

@@ -3,10 +3,6 @@ locals {
   aws_region  = var.aws_region
 }
 
-resource "random_id" "random" {
-  byte_length = 20
-}
-
 module "base" {
   source = "../base"
 
@@ -27,9 +23,9 @@ module "runners" {
   }
 
   github_app = {
-    key_base64     = var.github_app.key_base64
-    id             = var.github_app.id
-    webhook_secret = random_id.random.hex
+    key_base64 = var.github_app.key_base64
+    id         = var.github_app.id
+    # webhook_secret = random_id.random.hex
   }
 
   # configure the block device mappings, default for Amazon Linux2
@@ -143,18 +139,6 @@ module "runners" {
   # kms_key_arn = aws_kms_key.github.arn
 }
 
-module "webhook_github_app" {
-  source     = "../../modules/webhook-github-app"
-  depends_on = [module.runners]
-
-  github_app = {
-    key_base64     = var.github_app.key_base64
-    id             = var.github_app.id
-    webhook_secret = random_id.random.hex
-  }
-  webhook_endpoint = module.runners.webhook.endpoint
-}
-
 # enable CMK instead of aws managed key for encryptions
 # resource "aws_kms_key" "github" {
 #   is_enabled = true

examples/default/outputs.tf

@@ -7,9 +7,3 @@ output "runners" {
 output "webhook_endpoint" {
   value = module.runners.webhook.endpoint
 }
-
-output "webhook_secret" {
-  sensitive = true
-  value     = random_id.random.hex
-}
-

main.tf

@@ -12,6 +12,17 @@ locals {
   runner_labels         = (var.runner_disable_default_labels == false) ? sort(concat(local.default_runner_labels, var.runner_extra_labels)) : var.runner_extra_labels
 
   ssm_root_path = var.ssm_paths.use_prefix ? "/${var.ssm_paths.root}/${var.prefix}" : "/${var.ssm_paths.root}"
+
+  github_app = merge(var.github_app, {
+    webhook_secret = var.github_app.webhook_secret != null ? var.github_app.webhook_secret : module.rotating_random[0].random.hex
+  })
+}
+
+module "rotating_random" {
+  count  = var.github_app.webhook_secret == null ? 1 : 0
+  source = "./modules/rotating-random"
+
+  rotation_days = var.github_app.webhook_secret_rotation_days
 }
 
 resource "random_string" "random" {
@@ -91,10 +102,18 @@ module "ssm" {
 
   kms_key_arn = var.kms_key_arn
   path_prefix = "${local.ssm_root_path}/${var.ssm_paths.app}"
-  github_app  = var.github_app
+  github_app  = local.github_app
   tags        = local.tags
 }
 
+module "webhook_github_app" {
+  count  = var.github_app.webhook_secret == null ? 1 : 0
+  source = "./modules/webhook-github-app"
+
+  github_app       = local.github_app
+  webhook_endpoint = "${module.webhook.gateway.api_endpoint}/${module.webhook.endpoint_relative_path}"
+}
+
 module "webhook" {
   source = "./modules/webhook"
 

modules/multi-runner/README.md

@@ -95,10 +95,12 @@ module "multi-runner" {
 |------|--------|---------|
 | <a name="module_ami_housekeeper"></a> [ami\_housekeeper](#module\_ami\_housekeeper) | ../ami-housekeeper | n/a |
 | <a name="module_instance_termination_watcher"></a> [instance\_termination\_watcher](#module\_instance\_termination\_watcher) | ../termination-watcher | n/a |
+| <a name="module_rotating_random"></a> [rotating\_random](#module\_rotating\_random) | ./../rotating-random | n/a |
 | <a name="module_runner_binaries"></a> [runner\_binaries](#module\_runner\_binaries) | ../runner-binaries-syncer | n/a |
 | <a name="module_runners"></a> [runners](#module\_runners) | ../runners | n/a |
 | <a name="module_ssm"></a> [ssm](#module\_ssm) | ../ssm | n/a |
 | <a name="module_webhook"></a> [webhook](#module\_webhook) | ../webhook | n/a |
+| <a name="module_webhook_github_app"></a> [webhook\_github\_app](#module\_webhook\_github\_app) | ./../webhook-github-app | n/a |
 
 ## Resources
 
@@ -130,8 +132,8 @@ module "multi-runner" {
 | <a name="input_enable_managed_runner_security_group"></a> [enable\_managed\_runner\_security\_group](#input\_enable\_managed\_runner\_security\_group) | Enabling the default managed security group creation. Unmanaged security groups can be specified via `runner_additional_security_group_ids`. | `bool` | `true` | no |
 | <a name="input_eventbridge"></a> [eventbridge](#input\_eventbridge) | Enable the use of EventBridge by the module. By enabling this feature events will be put on the EventBridge by the webhook instead of directly dispatching to queues for scaling. | <pre>object({<br/>    enable        = optional(bool, true)<br/>    accept_events = optional(list(string), [])<br/>  })</pre> | `{}` | no |
 | <a name="input_ghes_ssl_verify"></a> [ghes\_ssl\_verify](#input\_ghes\_ssl\_verify) | GitHub Enterprise SSL verification. Set to 'false' when custom certificate (chains) is used for GitHub Enterprise Server (insecure). | `bool` | `true` | no |
-| <a name="input_ghes_url"></a> [ghes\_url](#input\_ghes\_url) | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB.However if you are using Github Enterprise Cloud with data-residency (ghe.com), set the endpoint here. Example - https://companyname.ghe.com| `string` | `null` | no |
-| <a name="input_github_app"></a> [github\_app](#input\_github\_app) | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`). | <pre>object({<br/>    key_base64     = string<br/>    id             = string<br/>    webhook_secret = string<br/>  })</pre> | n/a | yes |
+| <a name="input_ghes_url"></a> [ghes\_url](#input\_ghes\_url) | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB. .However if you are using Github Enterprise Cloud with data-residency (ghe.com), set the endpoint here. Example - https://companyname.ghe.com\| | `string` | `null` | no |
+| <a name="input_github_app"></a> [github\_app](#input\_github\_app) | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`)."<br/><br/>    If `webhook_secret` is not set, a random secret will be generated and stored in SSM. The secret is used to validate the webhook events. If you want to use your own secret, set the `webhook_secret` parameter.<br/>    When the secret is managed by the module, it will be rotated every `webhook_secret_rotation_days` days. | <pre>object({<br/>    key_base64                   = string<br/>    id                           = string<br/>    webhook_secret               = optional(string)<br/>    webhook_secret_rotation_days = optional(number, 30)<br/>  })</pre> | n/a | yes |
 | <a name="input_instance_profile_path"></a> [instance\_profile\_path](#input\_instance\_profile\_path) | The path that will be added to the instance\_profile, if not set the environment name will be used. | `string` | `null` | no |
 | <a name="input_instance_termination_watcher"></a> [instance\_termination\_watcher](#input\_instance\_termination\_watcher) | Configuration for the spot termination watcher lambda function. This feature is Beta, changes will not trigger a major release as long in beta.<br/><br/>`enable`: Enable or disable the spot termination watcher.<br/>`memory_size`: Memory size linit in MB of the lambda.<br/>`s3_key`: S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas.<br/>`s3_object_version`: S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket.<br/>`timeout`: Time out of the lambda in seconds.<br/>`zip`: File location of the lambda zip file. | <pre>object({<br/>    enable = optional(bool, false)<br/>    features = optional(object({<br/>      enable_spot_termination_handler              = optional(bool, true)<br/>      enable_spot_termination_notification_watcher = optional(bool, true)<br/>    }), {})<br/>    memory_size       = optional(number, null)<br/>    s3_key            = optional(string, null)<br/>    s3_object_version = optional(string, null)<br/>    timeout           = optional(number, null)<br/>    zip               = optional(string, null)<br/>  })</pre> | `{}` | no |
 | <a name="input_key_name"></a> [key\_name](#input\_key\_name) | Key pair name | `string` | `null` | no |

modules/multi-runner/main.tf

@@ -16,6 +16,17 @@ locals {
   unique_os_and_arch                   = { for i, v in local.tmp_distinct_list_unique_os_and_arch : "${v.os_type}_${v.architecture}" => v }
 
   ssm_root_path = "/${var.ssm_paths.root}/${var.prefix}"
+
+  github_app = merge(var.github_app, {
+    webhook_secret = var.github_app.webhook_secret != null ? var.github_app.webhook_secret : module.rotating_random[0].random.hex
+  })
+}
+
+module "rotating_random" {
+  count  = var.github_app.webhook_secret == null ? 1 : 0
+  source = "./../rotating-random"
+
+  rotation_days = var.github_app.webhook_secret_rotation_days
 }
 
 resource "random_string" "random" {

modules/multi-runner/ssm.tf

@@ -3,6 +3,6 @@ module "ssm" {
 
   kms_key_arn = var.kms_key_arn
   path_prefix = "${local.ssm_root_path}/${var.ssm_paths.app}"
-  github_app  = var.github_app
+  github_app  = local.github_app
   tags        = local.tags
 }

modules/multi-runner/variables.tf

@@ -1,9 +1,15 @@
 variable "github_app" {
-  description = "GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`)."
+  description = <<EOF
+    GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`)."
+
+    If `webhook_secret` is not set, a random secret will be generated and stored in SSM. The secret is used to validate the webhook events. If you want to use your own secret, set the `webhook_secret` parameter.
+    When the secret is managed by the module, it will be rotated every `webhook_secret_rotation_days` days.
+  EOF
   type = object({
-    key_base64     = string
-    id             = string
-    webhook_secret = string
+    key_base64                   = string
+    id                           = string
+    webhook_secret               = optional(string)
+    webhook_secret_rotation_days = optional(number, 30)
   })
 }