Pull Request Testing Interface

Pull Request Testing

Changes to the Trilinos develop branch must be made through Pull Requests. Prior to applying changes to the develop branch, the Trilinos Project supports a pull request testing feature. Pull request testing checks a small number of builds to support basic stability and provide developers with confidence that their changes are not breaking key configurations. Pull request testing is not comprehensive, and is not meant to guarantee broad portability. Also, pull request testing is only supported for pull requests submitted against the develop branch of Trilinos. Pull requests to all other branches are ignored.

The Autotester tool that drives pull request testing for Trilinos is scheduled to run every 20 minutes, and does so unless another pull request being tested blocks it from doing so, in which case, the tester will run again after the current test run has completed. The tool has limited ability to run in parallel. Prior to being tested, a pull request must pass pre-test inspection. This inspection can be satisfied by either

• the pull request submitter being a member of the Trilinos GitHub organization

or

• a member of the Trilinos GitHub organization approving the pull request

There is also a pre-merge inspection. This inspection can only be satisfied through a member of the Trilinos GitHub organization approving the pull request (with no other members requesting changes), or through special exception.

The Autotester schedules Jenkins jobs to confirm the submitted code will configure, build, and test using broad but fairly vanilla configurations and implements the Pull Request Autotester inspection. All builds set standard to C++14. The current set of builds performed on each pull request includes:

• GCC 8.3.0 with openmpi-1.10.1 and pthreads
  • Complex double support
  • Warnings as errors flag set
  • OpenMP = ON
  • Tpetra INTA Serial = ON
  • Build Shared Libs = OFF
  • C++17
• GCC 7.2.0 Serial (no MPI) and pthreads
  • Fortran disabled
  • Complex double support
  • Warnings as errors flag set
• GCC 7.2.0 Debug with openmpi-1.10.1 and pthreads
  • Complex double support
  • Warnings as errors flag set
• Intel 17.0.1 with MPICH 3.2 and pthreads
  • Tpetra_INST_INT_INT=ON set
• CUDA 9.2
  • Some packages and tests disabled
• Clang 10.0.0
  • Some tests disabled

Note that these are meant to assure that the basic work of other developers can continue, not to replace the fuller SQA cycle represented by the nightly dashboard and release process.

When the three status tests listed above (pre-test inspection, autotester inspection, and pre-merge inspection) all pass, and GitHub indicates that the merge can be done without conflict, the merge button will turn green as an indicator the PR is ready to be merged. Note that having the AT:WIP label set will stop the autotester from proceeding to this check.

Understanding and controlling what the autotester does

• Turn off testing - Add the AT:WIP (Work In Progress) label to the PR. No testing will occur until it is removed.

• Retest - Add the AT:RETEST label to the PR. The autotester will remove this label when it schedules the next build and test.

• Automerge when testing completes - Add the AT:AUTOMERGE label to the PR. Once all required checks have passed, the autotester will merge the PR without further user intervention.

• Prevent stale PRs from being merged - If testing passed, but it has been at least 6 days since the PR was tested, the autotester will apply the AT:STALE label to the PR. This will prevent the PR from being merged until the PR is tested again. Apply the AT:RETEST to request that a stale PR be retested.

Finding the errors from an autotest build

The Jenkins builds from the autotester report to the CDash page under the Pull Request Group. These builds have names of the form PR-<pr-id>-test-Trilinos_pullrequest_<build-name>-<build-num>, where <pr-id> and <build-num> are shown in the PR comments created by the autotester when testing is started and when testing is completed.

Note that the CDash entry you are looking in the Pull Request Group for may not be on the current date. A full query can be constructed using the PR number and all dates before today similar to this query - just replace the 99999 (the <pr-id>) with your PR number.

Build Options

All builds now set standard to C++14

	GCC 7.2.0	GCC 7.2.0 serial	GCC 7.2.0 debug	GCC 8.3.0	Intel 17.0.1	Intel 19.0.5	CUDA 9.2.88 *1	CUDA 10.1.105 *1	CUDA 10.1.105 UVM *1	Clang 7.0.1	Clang 9.0.0	Clang 10.0.0
Used for Pull Requests		X	X	X	X			X				X
Used for develop to master merge builds	X	X	X	X	X	X	X	X	X	X	X	X
MPI_EXEC_PRE_NUMPROCS_FLAGS = --bind to none	set	set	set	set						set	set	set
Complex Double	Enabled	Enabled	Enabled	Enabled
Warning as Errors	Global	Global	Global	Global	Partial *2
FORTRAN		Disabled
Tpetra_INST_INT_INT					ON
OpenMP				ON
Tpetra INTA Serial				ON
Build Shared Libs				OFF
UVM									OFF
Tests Disabled *see full build options below	1	1	3	1	4	4	27	11	See Below	11	11	11

*¹ There are many options necessary for the CUDA builds.
    None are specific to any additional testing beyond CUDA.
    See full setting below for details.

*² Intel WError set only for Intrepid2, Pike, & Tempus

For full build options, see the settings files directly:

• Common Settings for parallel builds
  • GCC 7.2.0
  • GCC 7.2.0 debug
  • GCC 8.3.0
  • Intel 17.0.1
  • CUDA 9.2
  • CUDA 10.1.105
  • CUDA 10.1.105 UVM [off]
  • Clang 7.0.1
  • Clang 9.0.0
  • Clang 10.0.0
• Common Settings for serial builds
  • GCC 7.2.0 serial

Notes on the PR Testing Interface is also kept on GitHub here.