feat: Enable skip of CI deployment tests

[ReneHezser]

Description

Allow testcases to be excluded from deplyoments by the CI deployment.

Azure/Azure-Verified-Modules#1796

Pipeline Reference

Pipeline

Type of Change

• Update to CI Environment or utilities (Non-module affecting changes)
• Azure Verified Module updates:
  • Bugfix containing backwards-compatible bug fixes, and I have NOT bumped the MAJOR or MINOR version in version.json:
    • Someone has opened a bug report issue, and I have included "Closes #{bug_report_issue_number}" in the PR description.
    • The bug was found by the module author, and no one has opened an issue to report it yet.
  • Feature update backwards compatible feature updates, and I have bumped the MINOR version in version.json.
  • Breaking changes and I have bumped the MAJOR version in version.json.
  • Update to documentation

Checklist

• I'm sure there are no other open Pull Requests for the same update/change
• I have run Set-AVMModule locally to generate the supporting module files.
• My corresponding pipelines / checks run clean and green without any errors or warnings

[eriqua]

Wondering if we could skip the whole job instead of singular steps

[ReneHezser]

Unfortunately, we can't. I moved the ignore check to the top of the steps, so it will "stop" earlier. Skipping a matrix job is not possible. We could implement another job before this one, to modify the tests-list input in order to only execute the non-skipped tests. But with that, the test would vanish completely from the list.

[eriqua]

In this case what about using the job title to provide that info, e.g. to add a skipped- prefix to the module test folder so that it gets rendered in the UI

[ReneHezser]

This is not possible, unfortunately. The skipping condition is only available in a step within the job. Not before :-(

[eriqua]

Looking forward to discussing, great work meanwhile!
The main suggestion I'd have for now is to add tests. At least one pipeline run with one deployment test skipped and one base case pipeline with no test skipped

[ReneHezser]

Thanks @eriqua.
I am not sure I understand your remark. You mean we should have a failed test before the test can be excluded?

[eriqua]

Hi @ReneHezser exactly, suggested full test coverage for CI updates before they are introduced.

• Static validation: we introduced 2 pester tests so we need:
  • module run with no test skipped --> should pass
  • module run where deployment tests are skipped as intended--> should pass
  • module run with one test skipped incorrectly (.e2eignore added in defaults or waf-aligned test) --> should fail pester test 1
  • module run with one test skipped incorrectly (.e2eignore is empty) --> should fail pester test 2