Pull Request Testing Interface

Pull Request Testing

Changes to the Trilinos develop branch must be made through Pull Requests. Prior to applying changes to the develop branch, the Trilinos Project supports a pull request testing feature. Pull request testing checks a small number of builds to support basic stability and provide developers with confidence that their changes are not breaking key configurations. Pull request testing is not comprehensive, and is not meant to guarantee broad portability. Also, pull request testing is only supported for pull requests submitted against the develop branch of Trilinos. Pull requests to all other branches are ignored.

The AutoTester tool that drives pull request testing for Trilinos is scheduled to run every 20 minutes, and does so unless another pull request being tested blocks it from doing so, in which case, the tester will run again after the current test run has completed. The tool has limited ability to run in parallel. Prior to being tested, a pull request must pass pre-test inspection. This inspection can be satisfied by either

• the pull request submitter being a member of the Trilinos GitHub organization

or

• a member of the Trilinos GitHub Developers team applying the AT: PRE-TEST INSPECTED label

There is also a pre-merge inspection. This inspection can only be satisfied through a member of the Trilinos GitHub organization approving the pull request (with no other members requesting changes), or through special exception.

The AutoTester schedules Jenkins jobs to confirm the submitted code will configure, build, and test using broad but fairly vanilla configurations and implements the Pull Request AutoTester inspection.

Note that these are meant to assure that the basic work of other developers can continue, not to replace the fuller SQA cycle represented by the nightly dashboard and release process.

When the three status tests listed above (pre-test inspection, AutoTester inspection, and pre-merge inspection) all pass, and GitHub indicates that the merge can be done without conflict, the merge button will turn green as an indicator the PR is ready to be merged. Note that having the AT: WIP label set will stop the AutoTester from proceeding to this check.

Understanding and controlling what the AutoTester does

• Turn off testing - Add the AT: WIP (Work In Progress) label to the PR or mark the PR as a Draft. No testing will occur until both the AT: WIP label and Draft marking are removed.

• Retest - Add the AT: RETEST label to the PR. The AutoTester will remove this label when it schedules the next build and test.

• Automerge when testing completes - Add the AT: AUTOMERGE label to the PR. Once all required checks have passed, the AutoTester will merge the PR without further user intervention.

• Prevent stale PRs from being merged - If testing passed, but it has been at least 6 days since the PR was tested, the AutoTester will apply the AT: STALE label to the PR. This will prevent the PR from being merged until the PR is tested again. Apply the AT: RETEST to request that a stale PR be retested.

Finding the errors from a Pull Request Build

The Jenkins builds from the AutoTester report to the CDash page under the Pull Request Group.

Note that the CDash entry you are looking in the Pull Request Group for may not be on the current date. A full query can be constructed using the PR number and all dates before today similar to this query - just replace the 99999 (the <pr-id>) with your PR number.

CDash Build Name Format

Old

PR-<pr-id>-test-Trilinos_pullrequest_<build-name>-<build-num>, where <pr-id> and <build-num> are shown in the PR comments created by the AutoTester when testing is started and when testing is completed.

New (transition in progress)

PR-<pr-id>-<toolchain>-<buildType>-<specialNameValues>-<build-num> where <toolchain> is the compiler/MPI (and possibly device compiler), <buildType> is roughly analagous to the CMAKE_BUILD_TYPE (e.g. release, debug), and <specialNameValues> is anything human-readable that indicates specialty about the given build (e.g. C++20 standard, UVM for CUDA, OpenMP, etc.).

Steps to reproduce a pull request build

See https://github.com/trilinos/Trilinos/wiki/Reproducing-PR-Testing-Errors.

Pull Request (${myBranch} -> develop) Tested Configurations

PR Build	Build Name	CDash Results	C++ Standard
GCC 8.3.0	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
GCC 8.3.0 serial	rhel8_sems-gnu-8.5.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
GCC 8.3.0	rhel8_sems-gnu-8.5.0-openmpi-4.1.6-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
Clang 11.0.1	rhel8_sems-clang-11.0.1-openmpi-4.0.5-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
Python 3	rhel8_aue-gnu-12.1.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework	CDash	N/A
CUDA 11	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
CUDA 11 uvm=on (no tests)	rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables	CDash	C++17
Intel 2021.3.0 (classic)	rhel8_sems-intel-2021.3-sems-openmpi-4.1.4_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17

Master Merge Process

Once changes are in the 'develop' branch, they need to pass through an additional layer of testing to make it to the 'master' (main) branch. This testing currently runs weekly, and if it passes, allows the current state on the develop branch to be merged onto the master branch. Guarding the master branch in this way allows for two additional pieces of software quality assurance. One is eliminating the possibility that two changes combine to form a bad state (even though they individually passed PR testing). The other is for additional builds/toolchains to be exercised that may be too expensive to run on an individual PR basis.

Master Merge (develop -> master) Tested Configurations

Dev2Master Build	Build Name	CDash Results	C++ Standard
GCC 8.3.0	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
GCC 8.3.0 serial	rhel7_sems-gnu-8.3.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
GCC 8.3.0	rhel7_sems-gnu-8.3.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
Python 3	rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework	CDash	N/A
Clang 11.0.1	rhel7_sems-clang-11.0.1-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
CUDA 11 uvm=off	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17
CUDA 11 uvm=on (no tests)	rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables	CDash	C++17
Intel 2021.3.0 (classic)	rhel7_sems-intel-2021.3-sems-openmpi-4.0.5_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables	CDash	C++17