Skip to content

Enable automation of the CodeQL CLI upgrade process #731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Oct 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 26 additions & 16 deletions .github/workflows/upgrade_codeql_dependencies.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,20 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Fetch CodeQL
env:
GITHUB_TOKEN: ${{ github.token }}
RUNNER_TEMP: ${{ runner.temp }}
run: |
cd $RUNNER_TEMP
gh release download "v${CODEQL_CLI_VERSION}" --repo https://github.com/github/codeql-cli-binaries --pattern codeql-linux64.zip
unzip -q codeql-linux64.zip
echo "$RUNNER_TEMP/codeql/" >> $GITHUB_PATH

- name: Install Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.9"

Expand All @@ -35,27 +45,27 @@ jobs:
run: |
python3 scripts/upgrade-codeql-dependencies/upgrade-codeql-dependencies.py --cli-version "$CODEQL_CLI_VERSION"

- name: Fetch CodeQL
env:
GITHUB_TOKEN: ${{ github.token }}
RUNNER_TEMP: ${{ runner.temp }}
run: |
cd $RUNNER_TEMP
gh release download "v${CODEQL_CLI_VERSION}" --repo https://github.com/github/codeql-cli-binaries --pattern codeql-linux64.zip
unzip -q codeql-linux64.zip

- name: Update CodeQL formatting based on new CLI version
env:
RUNNER_TEMP: ${{ runner.temp }}
run: |
find cpp \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" $RUNNER_TEMP/codeql/codeql query format --in-place
find c \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" $RUNNER_TEMP/codeql/codeql query format --in-place
find cpp \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" codeql query format --in-place
find c \( -name '*.ql' -or -name '*.qll' \) -print0 | xargs -0 --max-procs "$XARGS_MAX_PROCS" codeql query format --in-place

- name: Create Pull Request
uses: peter-evans/create-pull-request@v3
uses: peter-evans/create-pull-request@5e914681df9dc83aa4e4905692ca88beb2f9e91f # v7.0.5
with:
title: "Upgrading `github/codeql` dependency to ${{ github.event.inputs.codeql_cli_version }}"
body: "This PR upgrades the CodeQL CLI version to ${{ github.event.inputs.codeql_cli_version }}."
title: "Upgrade `github/codeql` dependency to ${{ github.event.inputs.codeql_cli_version }}"
body: |
This PR upgrades the CodeQL CLI version to ${{ github.event.inputs.codeql_cli_version }}.

## CodeQL dependency upgrade checklist:

- [ ] Confirm the code has been correctly reformatted according to the new CodeQL CLI.
- [ ] Identify any CodeQL compiler warnings and errors, and update queries as required.
- [ ] Validate that the `github/codeql` test cases succeed.
- [ ] Address any CodeQL test failures in the `github/codeql-coding-standards` repository.
- [ ] Validate performance vs pre-upgrade, using /test-performance
commit-message: "Upgrading `github/codeql` dependency to ${{ github.event.inputs.codeql_cli_version }}"
delete-branch: true
branch: "codeql/upgrade-to-${{ github.event.inputs.codeql_cli_version }}"
43 changes: 25 additions & 18 deletions docs/development_handbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,46 +496,53 @@ There are two external dependencies required for running the coding standards qu

For the purpose of this repository, and any tool qualification, we consider these external dependencies to be "black boxes" which require verification when upgrading.

To (a) clearly specify the supported versions of these external dependencies and to (b) enable automation around them, the repository contains a `supported_codeql_configs.json` which lists the sets of supported configurations. There are four fields:
To (a) clearly specify the supported versions of these external dependencies and to (b) enable automation around them, the repository contains a `supported_codeql_configs.json` which lists the sets of supported configurations under the `supported_environments` property. There are three fields:

- `codeql_cli` - this is the plain version number of the supported CodeQL CLI, e.g. `2.6.3`.
- `codeql_standard_library` - this is the name of a tag on the `github.com/github/codeql` repository. The tag should be compatible with the CodeQL CLI given above. This would typically use the `codeql-cli/v<version-number>` tag for the release, although any tag which is compatible is allowed.
- `codeql_cli_bundle` - (optional) - if present, describes the CodeQL CLI bundle version that is compatible. The bundle should include precisely the CodeQL CLI version and CodeQL Standard Library versions specified in the two mandatory fields.
- `ghes` - (optional) - if present describes the GitHub Enterprise Server release whose integrated copy of the CodeQL Action points to the CodeQL CLI bundle specified in the `codeql_cli_bundle` field.

#### Upgrading external dependencies

To upgrade the CodeQL external dependencies:

1. Determine appropriate versions of the CodeQL CLI and `github/codeql` repository, according to the release schedule and customer demands.
2. Determine if there is a compatible CodeQL CLI bundle version by looking at the releases specified at [CodeQL Action releases](https://github.com/github/codeql-action/releases). The bundle always includes the standard library at the version specified by the `codeql-cli/v<version-number>` tag in the `github/codeql` repository.
3. If you find a compatible CodeQL CLI bundle, determine whether that bundle was released in a GitHub Enterprise server release, by inspecting the `defaults.json` file at https://github.com/github/codeql-action/blob/main/lib/defaults.json#L2 for the CodeQL Action submitted with
4. Populated the `supported_codeql_configs.json` file with the given values, ensuring to delete the optional fields if they are not populated.
5. Submit a Pull Request to the `github/codeql-coding-standards` repository with the title `Upgrade `github/codeql` dependency to <insert codeql_standard_library value>`. Use this template for the description, filling :

```md
This PR updates the `supported_codeql_configs.json` file to target:
If all components are being upgraded to a consistent veresion (e.g. CodeQL CLI v2.15.5, with `github/codeql` tag `codeql-cli/v2.15.5` and bundle `codeql-cli-bundle-v2.15.5`) then the following process can be used:

1. Run the [upgrade_codeql_dependencies.yml](./github/workflows/upgrade_codeql_dependencies.yml) workflow, with the plain version number, e.g. `2.15.5`. This will:
- Download the specified version of the CodeQL CLI
- Run the [upgrade-codeql-dependencies.py](scripts/release/upgrade-codeql-dependencies.py) script, which
- Validates the version selected exists in all relevant places
- Updates the `supported_codeql_configs.json` file.
- Updates each `qlpack.yml` in the repository with an appropriate value for the `codeql/cpp-all` pack, consistent with the selected CodeQL CLI version.
- Updates each `codeql-lock.yml` file to upgrade to the new version.
2. Follow the dependency upgrade checklist, confirming each step. The `.github/workflows/standard_library_upgrade_tests.yml` will trigger automation for running the `github/codeql` unit tests with the appropriate CLI version.
3. Once all the automate tests have passed, and the checklist is complete, the PR can be merged.
4. An internal notification should be shared with the development team.

- CodeQL CLI <codeql_cli>
- CodeQL Standard Library <codeql_standard_library>
- GHES <ghes>
- CodeQL CLI Bundle <date_of_bundle>
If the upgrade is of mismatched versions you will need to manually create the upgrade following this process:

<EITHER:This should match the versions of CodeQL deployed with GitHub Enterprise Server <ghes>>
<OR: This does not match any released version of GitHub Enterprise Server.>
1. Populate the `supported_codeql_configs.json` file with the given values, ensuring to delete the optional fields if they are not populated.
2. Submit a Pull Request to the `github/codeql-coding-standards` repository with the title `Upgrade `github/codeql` dependency to <insert codeql_standard_library value>`. Use this template for the description, filling:

```md
This PR updates the `supported_codeql_configs.json` file to target CodeQL CLI <codeql_cli>.

## CodeQL dependency upgrade checklist:

- [ ] Reformat our CodeQL using the latest version (if required)
- [ ] Confirm the code has been correctly reformatted according to the new CodeQL CLI.
- [ ] Identify any CodeQL compiler warnings and errors, and update queries as required.
- [ ] Validate that the `github/codeql` test cases succeed.
- [ ] Address any CodeQL test failures in the `github/codeql-coding-standards` repository.
- [ ] Validate performance vs pre-upgrade
- [ ] Validate performance vs pre-upgrade, using /test-performance
```

6. Follow the dependency upgrade checklist, confirming each step. The `.github/workflows/standard_library_upgrade_tests.yml` will trigger automation for running the `github/codeql` unit tests with the appropriate CLI version.
7. Once all the automate tests have passed, and the checklist is complete, the PR can be merged.
8. An internal notification should be shared with the development team.
3. Follow the dependency upgrade checklist, confirming each step. The `.github/workflows/standard_library_upgrade_tests.yml` will trigger automation for running the `github/codeql` unit tests with the appropriate CLI version.
4. Once all the automate tests have passed, and the checklist is complete, the PR can be merged.
5. An internal notification should be shared with the development team.


### Release process

Expand Down
1 change: 1 addition & 0 deletions scripts/upgrade-codeql-dependencies/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ idna==3.4
requests==2.31.0
semantic-version==2.10.0
urllib3==1.26.18
pyyaml==6.0.1
48 changes: 41 additions & 7 deletions scripts/upgrade-codeql-dependencies/upgrade-codeql-dependencies.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,23 @@
import json
import requests
from typing import Optional, Dict, List
from typing import Optional, Dict, List, Tuple
from semantic_version import Version
from pathlib import Path
import yaml

SCRIPT_PATH = Path(__file__)
SUPPORTED_VERSIONS_PATH = SCRIPT_PATH.parent.parent.parent / "supported_codeql_configs.json"
CODING_STANDARDS_ROOT = SCRIPT_PATH.parent.parent.parent
SUPPORTED_VERSIONS_PATH = CODING_STANDARDS_ROOT / "supported_codeql_configs.json"

def get_compatible_stdlib(version: Version) -> Optional[str]:
def get_compatible_stdlib(version: Version) -> Optional[Tuple[str, str]]:
tag = f"codeql-cli/v{version}"
response = requests.get(f"https://raw.githubusercontent.com/github/codeql/{tag}/cpp/ql/lib/qlpack.yml")

if response.status_code == 200:
return tag
# Parse the qlpack.yml returned in the response as a yaml file to read the version property
qlpack = yaml.safe_load(response.text)
if qlpack is not None and "version" in qlpack:
return (tag, qlpack["version"])
return None

def get_compatible_bundle(version: Version, token: str) -> Optional[str]:
Expand All @@ -30,15 +35,17 @@ def get_compatible_bundle(version: Version, token: str) -> Optional[str]:
def main(cli_version : str, github_token: str) -> None:
try:
parsed_cli_version = Version(cli_version)
compatible_stdlib = get_compatible_stdlib(parsed_cli_version)
if compatible_stdlib is None:
compatible_stdlib_return = get_compatible_stdlib(parsed_cli_version)
if compatible_stdlib_return is None:
print(f"Unable to find compatible standard library for: {parsed_cli_version}")
exit(1)
compatible_bundle = get_compatible_bundle(parsed_cli_version, github_token)
if compatible_bundle is None:
print(f"Unable to find compatible bundle for: {parsed_cli_version}")
exit(1)

compatible_stdlib_tag, compatible_stdlib_version = compatible_stdlib_return

with SUPPORTED_VERSIONS_PATH.open("r") as f:
supported_versions = json.load(f)

Expand All @@ -49,10 +56,37 @@ def main(cli_version : str, github_token: str) -> None:
supported_env = supported_envs[0]
supported_env["codeql_cli"] = str(parsed_cli_version)
supported_env["codeql_cli_bundle"] = compatible_bundle
supported_env["codeql_standard_library"] = compatible_stdlib
supported_env["codeql_standard_library"] = compatible_stdlib_tag

with SUPPORTED_VERSIONS_PATH.open("w") as f:
json.dump(supported_versions, f, indent=2)

# Find every qlpack.yml file in the repository
qlpack_files = list(CODING_STANDARDS_ROOT.rglob("qlpack.yml"))
# Filter out any files that are in a hidden directory
qlpack_files = [f for f in qlpack_files if not any(part for part in f.parts if part.startswith("."))]

# Update the "codeql/cpp-all" entries in the "dependencies" property in every qlpack.yml file
for qlpack_file in qlpack_files:
with qlpack_file.open("r") as f:
qlpack = yaml.safe_load(f)
print("Updating dependencies in " + str(qlpack_file))
if "codeql/cpp-all" in qlpack["dependencies"]:
qlpack["dependencies"]["codeql/cpp-all"] = compatible_stdlib_version
with qlpack_file.open("w") as f:
yaml.safe_dump(qlpack, f, sort_keys=False)

# Call CodeQL to update the lock files by running codeql pack upgrade
# Note: we need to do this after updating all the qlpack files,
# otherwise we may get dependency resolution errors
# Note: we need to update all qlpack files, because they may
# transitively depend on the packs we changed
for qlpack_file in qlpack_files:
qlpack = qlpack_file.parent
print("Updating lock files for " + str(qlpack))
os.system(f"codeql pack upgrade {qlpack}")


except ValueError as e:
print(e)
exit(1)
Expand Down
Loading