feat: Optimized deployment resolution runtime to reduce unnecessary w…

…ait time (#2461) ## Description - Changed the way the removal logic tries to resolve deployment names to their actual resources - Up until now the logic was as follows - One deployment name after another (up to 3 as per 3 retries) where passed into the script - The script would then, for each deploymentname respectively search for the deployment - If it didn't find it, it would retry after one minute up to 40 times - This meant, for 3 failed deployments that fail in a way that they don't create a deployment (e.g. in case the template failed already the initial validation, it would wait 3 x 40 minutes = 120 minutes - This, updated approach works as follows - All 3 deployment names are passed into the script - For each round of trying, it searches for all 3 deployment names - If at least one isn't found, it will try again, as before after one minute to a maximum of 40 times - Each found name will be skript by the retries - This way, the script in its entirety only works 40 minutes in total and as such will make sure it will definitely wait a maximum of 40 minutes for each deployment to show up > Note: To test a failing deployment, I used a long-running deployment script which I would maniupulate in Azure during deployment, and also delete one of the retries before the removal would start. This is why in the log you can see that the removal logic finds attempts 2 & 3 but not 1, which is what was intended. >```bicep > // In dependencies.bicep of WAF-aligned test >resource deploymentScript 'Microsoft.Resources/deploymentScripts@2023-08-01' = { > name: 'testScript' > location: location > identity: { > type: 'UserAssigned' > userAssignedIdentities: { > '${managedIdentity.id}': {} > } > } > kind: 'AzurePowerShell' > properties: { > azPowerShellVersion: '9.7' > retentionInterval: 'P1D' > scriptContent: 'Start-Sleep 180' > } >} >``` ## Pipeline Reference  | Pipeline | | -------- | | [![avm.res.network.virtual-network](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.network.virtual-network.yml/badge.svg?branch=users%2Falsehr%2FdeploymentWaitOptimization&event=workflow_dispatch)](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.network.virtual-network.yml) (failed deployment test) | | [![avm.res.key-vault.vault](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.key-vault.vault.yml/badge.svg?branch=users%2Falsehr%2FdeploymentWaitOptimization&event=workflow_dispatch)](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.key-vault.vault.yml) (successful deployment test) | Test run with a faulty module (VNET): https://github.com/Azure/bicep-registry-modules/actions/runs/9613612546 Test run with a working module (KeyVault): https://github.com/Azure/bicep-registry-modules/actions/runs/9536144913 ## Type of Change  - [x] Update to CI Environment or utilities (Non-module affecting changes) - [ ] Azure Verified Module updates: - [ ] Bugfix containing backwards-compatible bug fixes, and I have NOT bumped the MAJOR or MINOR version in `version.json`: - [ ] Someone has opened a bug report issue, and I have included "Closes #{bug_report_issue_number}" in the PR description. - [ ] The bug was found by the module author, and no one has opened an issue to report it yet. - [ ] Feature update backwards compatible feature updates, and I have bumped the MINOR version in `version.json`. - [ ] Breaking changes and I have bumped the MAJOR version in `version.json`. - [ ] Update to documentation ## Example log: **Successful deployments run** ``` VERBOSE: Invoke task with VERBOSE: { "SubscriptionId": "***", "ManagementGroupId": "***", "TemplateFilePath": "/home/runner/work/bicep-registry-modules/bicep-registry-modules/avm/res/key-vault/vault/tests/e2e/waf-aligned/main.test.bicep", "DeploymentNames": "a-r-kv-v-waf-aligned-t1-20240616T1206450259Z" } VERBOSE: Handling resource removal with deployment names VERBOSE: - a-r-kv-v-waf-aligned-t1-20240616T1206450259Z VERBOSE: Total number of deployment target resources after fetching deployments [18] VERBOSE: Total number of deployment target resources after pre-filtering (duplicates) & ordering items [18] VERBOSE: Total number of deployment target resources after formatting items [18] VERBOSE: Total number of deployments after filtering all dependency resources [18] VERBOSE: Total number of deployments after final ordering of resources [18] VERBOSE: - Remove [/subscriptions/***/resourceGroups/dep-***-keyvault.vaults-kvvwaf-rg/providers/Microsoft.KeyVault/vaults/***kvvwaf002/providers/Microsoft.Authorization/locks/myCustomLockName] ``` ## Example log: **Failed deployments run** ``` VERBOSE: Invoke task with VERBOSE: { "ManagementGroupId": "***", "SubscriptionId": "***", "DeploymentNames": [ "a-r-n-vn-waf-aligned-t1-20240621T1206370696Z", "a-r-n-vn-waf-aligned-t2-20240621T1206433769Z", "a-r-n-vn-waf-aligned-t3-20240621T1206013450Z" ], "TemplateFilePath": "/home/runner/work/bicep-registry-modules/bicep-registry-modules/avm/res/network/virtual-network/tests/e2e/waf-aligned/main.test.bicep" } VERBOSE: Handling resource removal with deployment names VERBOSE: - a-r-n-vn-waf-aligned-t1-20240621T1206370696Z VERBOSE: - a-r-n-vn-waf-aligned-t2-20240621T1206433769Z VERBOSE: - a-r-n-vn-waf-aligned-t3-20240621T1206013450Z VERBOSE: Found & resolved deployment [a-r-n-vn-waf-aligned-t2-20240621T1206433769Z] VERBOSE: Found & resolved deployment [a-r-n-vn-waf-aligned-t3-20240621T1206013450Z] VERBOSE: No deployment found by name(s) [@{Name=a-r-n-vn-waf-aligned-t1-20240621T1206370696Z}] in scope [subscription]. Retrying in [60] seconds [1/40] VERBOSE: No deployment found by name(s) [@{Name=a-r-n-vn-waf-aligned-t1-20240621T1206370696Z}] in scope [subscription]. Retrying in [60] seconds [2/40] ```
Azure · Jul 22, 2024 · b2e88e7 · b2e88e7
1 parent 0c380c4
commit b2e88e7
Show file tree

Hide file tree

Showing 2 changed files with 54 additions and 30 deletions.
diff --git a/...ities/pipelines/e2eValidation/resourceRemoval/helper/Get-DeploymentTargetResourceList.ps1 b/...ities/pipelines/e2eValidation/resourceRemoval/helper/Get-DeploymentTargetResourceList.ps1
@@ -188,7 +188,8 @@ function Get-DeploymentTargetResourceList {
         [string] $ManagementGroupId,
 
         [Parameter(Mandatory = $true)]
-        [string] $Name,
+        [Alias('Name', 'Names', 'DeploymentName')]
+        [string[]] $DeploymentNames,
 
         [Parameter(Mandatory = $true)]
         [ValidateSet(
@@ -207,31 +208,57 @@ function Get-DeploymentTargetResourceList {
     )
 
     $searchRetryCount = 1
-    do {
-        $innerInputObject = @{
-            Name        = $name
-            Scope       = $scope
-            ErrorAction = 'SilentlyContinue'
+    $resourcesToRemove = @()
+    $deploymentNameObjects = $DeploymentNames | ForEach-Object {
+        @{
+            Name     = $_
+            Resolved = $false
         }
-        if (-not [String]::IsNullOrEmpty($resourceGroupName)) {
-            $innerInputObject['resourceGroupName'] = $resourceGroupName
-        }
-        if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
-            $innerInputObject['ManagementGroupId'] = $ManagementGroupId
+    }
+
+    do {
+        foreach ($deploymentNameObject in $deploymentNameObjects) {
+
+            if ($deploymentNameObject.Resolved) {
+                # Skip further invocations for this deployment name if deployment was already found
+                continue
+            }
+
+            $innerInputObject = @{
+                Name        = $deploymentNameObject.Name
+                Scope       = $scope
+                ErrorAction = 'SilentlyContinue'
+            }
+            if (-not [String]::IsNullOrEmpty($resourceGroupName)) {
+                $innerInputObject['resourceGroupName'] = $resourceGroupName
+            }
+            if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
+                $innerInputObject['ManagementGroupId'] = $ManagementGroupId
+            }
+
+            [array]$targetResources = Get-DeploymentTargetResourceListInner @innerInputObject
+            if ($targetResources.Count -gt 0) {
+                Write-Verbose ('Found & resolved deployment [{0}]' -f $deploymentNameObject.Name) -Verbose
+                $deploymentNameObject.Resolved = $true
+                $resourcesToRemove += $targetResources
+            }
         }
-        [array]$targetResources = Get-DeploymentTargetResourceListInner @innerInputObject
-        if ($targetResources) {
+
+        # Break check
+        if ($deploymentNameObjects.Resolved -notcontains $false) {
             break
         }
-        Write-Verbose ('No deployment found by name [{0}] in scope [{1}]. Retrying in [{2}] seconds [{3}/{4}]' -f $name, $scope, $searchRetryInterval, $searchRetryCount, $searchRetryLimit) -Verbose
+
+        $remainingDeploymentNames = ($deploymentNameObjects | Where-Object { -not $_.Resolved }).Name
+        Write-Verbose ('No deployment found by name(s) [{0}] in scope [{1}]. Retrying in [{2}] seconds [{3}/{4}]' -f ($remainingDeploymentNames -join ', '), $scope, $searchRetryInterval, $searchRetryCount, $searchRetryLimit) -Verbose
         Start-Sleep $searchRetryInterval
         $searchRetryCount++
     } while ($searchRetryCount -le $searchRetryLimit)
 
-    if (-not $targetResources) {
-        Write-Warning "No deployment target resources found for [$name]"
+    if (-not $resourcesToRemove) {
+        Write-Warning ('No deployment target resources found for [{0}]' -f ($DeploymentNames -join ', '))
         return @()
     }
 
-    return $targetResources
+    return $resourcesToRemove
 }
diff --git a/avm/utilities/pipelines/e2eValidation/resourceRemoval/helper/Remove-Deployment.ps1 b/avm/utilities/pipelines/e2eValidation/resourceRemoval/helper/Remove-Deployment.ps1
@@ -90,20 +90,17 @@ function Remove-Deployment {
 
             # Fetch deployments
             # =================
-
-            foreach ($deploymentName in $DeploymentNames) {
-                $deploymentsInputObject = @{
-                    Name  = $deploymentName
-                    Scope = $deploymentScope
-                }
-                if (-not [String]::IsNullOrEmpty($ResourceGroupName)) {
-                    $deploymentsInputObject['resourceGroupName'] = $ResourceGroupName
-                }
-                if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
-                    $deploymentsInputObject['ManagementGroupId'] = $ManagementGroupId
-                }
-                $deployedTargetResources += Get-DeploymentTargetResourceList @deploymentsInputObject
+            $deploymentsInputObject = @{
+                DeploymentNames = $DeploymentNames
+                Scope           = $deploymentScope
+            }
+            if (-not [String]::IsNullOrEmpty($ResourceGroupName)) {
+                $deploymentsInputObject['resourceGroupName'] = $ResourceGroupName
+            }
+            if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
+                $deploymentsInputObject['ManagementGroupId'] = $ManagementGroupId
             }
+            $deployedTargetResources += Get-DeploymentTargetResourceList @deploymentsInputObject
 
             if ($deployedTargetResources.Count -eq 0) {
                 throw 'No deployment target resources found.'