Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable skip of CI deployment tests #4203

Draft
wants to merge 33 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
475bfce
check for a file .e2eignore in test folders
ReneHezser Jan 14, 2025
3c848bd
LF
ReneHezser Jan 14, 2025
219f6ff
formats output
ReneHezser Jan 14, 2025
17e6b6d
ensure the defaults and waf-aligned test for res modules
ReneHezser Jan 15, 2025
5aec7c1
require core team approval
ReneHezser Jan 17, 2025
8a79bc3
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Jan 17, 2025
c1171ec
cleanup
ReneHezser Jan 17, 2025
8e4dc0d
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Jan 20, 2025
c382179
ensure the .e2eignore file contains text
ReneHezser Jan 24, 2025
b28e053
test
ReneHezser Jan 24, 2025
c362c71
revert test
ReneHezser Jan 24, 2025
f547f3c
skip in deployment task
ReneHezser Jan 24, 2025
870bc34
test job output result
ReneHezser Jan 24, 2025
e88659b
test
ReneHezser Jan 24, 2025
8f47a3d
errorhandling
ReneHezser Jan 24, 2025
1ab729c
error test
ReneHezser Jan 24, 2025
01a1201
fixes
ReneHezser Jan 24, 2025
eb492e3
testing
ReneHezser Jan 27, 2025
47e9282
testing
ReneHezser Jan 27, 2025
9e5b97c
test
ReneHezser Jan 27, 2025
1630bcb
fix for test
ReneHezser Jan 27, 2025
6f08745
skip more steps, if the deployment test is skipped
ReneHezser Jan 27, 2025
d5e9e2e
cleanup
ReneHezser Jan 27, 2025
73e22f7
Merge branch 'main' into enable-skip-ci-deployment-tests
ReneHezser Jan 27, 2025
57bbbe0
Merge branch 'main' into enable-skip-ci-deployment-tests
ReneHezser Jan 28, 2025
a6df0eb
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Jan 28, 2025
e5898d3
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Jan 29, 2025
77ffdb1
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Jan 30, 2025
cf3de24
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Feb 4, 2025
f85f137
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Feb 7, 2025
4ff05fb
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Feb 10, 2025
d3d77e5
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Feb 12, 2025
a0dc290
Merge branch 'Azure:main' into enable-skip-ci-deployment-tests
ReneHezser Feb 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -180,3 +180,4 @@
/avm/res/web/static-site/ @Azure/avm-res-web-staticsite-module-owners-bicep @Azure/avm-module-reviewers-bicep
/avm/utl/types/avm-common-types/ @Azure/avm-utl-types-avmcommontypes-module-owners-bicep @Azure/avm-module-reviewers-bicep
*avm.core.team.tests.ps1 @Azure/avm-core-team-technical-bicep
*.e2eignore @Azure/avm-core-team-technical-bicep
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to discussing, great work meanwhile!
The main suggestion I'd have for now is to add tests. At least one pipeline run with one deployment test skipped and one base case pipeline with no test skipped

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @eriqua.
I am not sure I understand your remark. You mean we should have a failed test before the test can be excluded?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ReneHezser exactly, suggested full test coverage for CI updates before they are introduced.

  • Static validation: we introduced 2 pester tests so we need:
    • module run with no test skipped --> should pass
    • module run where deployment tests are skipped as intended--> should pass
    • module run with one test skipped incorrectly (.e2eignore added in defaults or waf-aligned test) --> should fail pester test 1
    • module run with one test skipped incorrectly (.e2eignore is empty) --> should fail pester test 2

Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,39 @@ runs:

# [Deployment validation] task(s)
# -------------------------------
- name: "Validate Test Execution"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we could skip the whole job instead of singular steps

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, we can't. I moved the ignore check to the top of the steps, so it will "stop" earlier. Skipping a matrix job is not possible. We could implement another job before this one, to modify the tests-list input in order to only execute the non-skipped tests. But with that, the test would vanish completely from the list.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case what about using the job title to provide that info, e.g. to add a skipped- prefix to the module test folder so that it gets rendered in the UI
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not possible, unfortunately. The skipping condition is only available in a step within the job. Not before :-(

uses: azure/powershell@v2
with:
azPSVersion: "latest"
inlineScript: |
# the first step is to check, if the deployment should be bypassed

# Resolve template file path
$moduleTestFilePath = Join-Path $env:GITHUB_WORKSPACE '${{ inputs.templateFilePath }}'

# Determine if $moduleTestFilePath is a directory or a file
if (Test-Path $moduleTestFilePath -PathType Container) {
$moduleTestFolderPath = $moduleTestFilePath
} else {
$moduleTestFolderPath = Split-Path $moduleTestFilePath
}

# Check if the deployment test should be bypassed
$passCiFilePath = Join-Path $moduleTestFolderPath '.e2eignore'
if (Test-Path $passCiFilePath) {
Write-Warning -Message "File '.e2eignore' exists in the folder: $moduleTestFolderPath"
$excludeReason = Get-Content $passCiFilePath
if ($excludeReason -ne $null) {
Write-Warning -Message "Reason for exclusion: $excludeReason"
}
Write-Output "skip_deployment_ci=true" >> $env:GITHUB_ENV
} else {
Write-Output "File '.e2eignore' does not exist in the folder: $moduleTestFolderPath" -Verbose
Write-Output "skip_deployment_ci=false" >> $env:GITHUB_ENV
}

- name: "Validate template file"
if: env.skip_deployment_ci == 'false'
uses: azure/powershell@v2
with:
azPSVersion: "latest"
Expand Down Expand Up @@ -293,6 +325,7 @@ runs:
# [Deployment execution] task(s)
# ------------------------------
- name: "Deploy template file"
if: env.skip_deployment_ci == 'false'
id: deploy_step
uses: azure/powershell@v2
with:
Expand Down Expand Up @@ -387,6 +420,7 @@ runs:
# [Post-Deployment test] task(s)
# ------------------------------
- name: "Run post-deployment Pester tests"
if: env.skip_deployment_ci == 'false'
id: pester_run_step
shell: pwsh
run: |
Expand Down Expand Up @@ -457,7 +491,7 @@ runs:
}

- name: "Output to GitHub job summaries"
if: steps.pester_run_step.outputs.formattedPesterResultsPath != ''
if: env.skip_deployment_ci == 'false' && steps.pester_run_step.outputs.formattedPesterResultsPath != ''
shell: pwsh
run: |
# Grouping task logs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,26 @@ Describe 'File/folder tests' -Tag 'Modules' {
$pathExisting | Should -Be $true -Because "path [$filePath] is expected to exist."
}
}

It '[<moduleFolderName>] Resource Modules must not skip the "defaults" or "waf-aligned" tests with a [` .e2eignore `] file.' -TestCases $topLevelModuleTestCases {

param(
[string] $moduleFolderName,
[string] $moduleFolderPath
)

if ($moduleFolderName -notmatch 'res/.*') {
Set-ItResult -Skipped -Because 'this test only applies to resource modules.'
return
}

$e2eTestFolderPathList = Get-ChildItem -Directory (Join-Path -Path $moduleFolderPath 'tests' 'e2e') | Where-Object { $_.Name -in 'defaults', 'waf-aligned' }
foreach ($e2eTestFolderPath in $e2eTestFolderPathList) {
$filePath = Join-Path -Path $e2eTestFolderPath '.e2eignore'
$pathExisting = Test-Path $filePath
$pathExisting | Should -Be $false -Because 'skipping this test is not allowed.'
}
}
}
}

Expand Down
10 changes: 10 additions & 0 deletions utilities/tools/Test-ModuleLocally.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -212,8 +212,18 @@ function Test-ModuleLocally {
# -------------------------
if ((Get-Item -Path $ModuleTestFilePath) -is [System.IO.DirectoryInfo]) {
$moduleTestFiles = (Get-ChildItem -Path $ModuleTestFilePath -File).FullName
$moduleTestFolderPath = $ModuleTestFilePath
} else {
$moduleTestFiles = @($ModuleTestFilePath)
$moduleTestFolderPath = Split-Path $ModuleTestFilePath
}

# Check if the deployment test should be bypassed
$passCiFilePath = Join-Path $moduleTestFolderPath '.e2eignore'
if (Test-Path $passCiFilePath) {
Write-Output "File '.e2eignore' exists in the folder: $moduleTestFolderPath"
# end here, as the test should be bypassed
return
}

# Construct Token Configuration Input
Expand Down
Loading