Codebuild batch and Omnibus job (aws#2245)

* ci: Create CodeBuild batch jobs for easy launching of all tests.
* Automation via troposphere not yet supported

.gitignore

@@ -34,3 +34,4 @@ test-deps/*
 CMakeCache.txt
 CMakeFiles/*
 .project
+./codebuild/spec/buildspec_*_batch.yml

codebuild/README.md

@@ -3,8 +3,10 @@
 #### Design
 
 - How does CodeBuild decide what to install/test ?
-   To match with Travis and minimize rewriting, the environment variables passed to the job
-   dictate what is installed and which tests get run.
+   Historically the environment variables passed to the job
+   dictate what is installed and which tests get run. CodeBuild has a pattern where environment
+   variables can be over-ridden by CloudWatch events or batch jobs, so in some cases the CodeBuild job definition
+   is generic or filled with placeholders (e.g. s2nFuzzScheduled).
 - Why not build docker images with the dependencies layered in ?
   This is the end goal: get tests running in CodeBuild first, then optimize the containers where it makes sense.
 
@@ -57,7 +59,7 @@ To bootstrap the CodeBuild jobs, the python script:
 ```
 
 If you are modifying an existing stack then a list of changes will be displayed and
-you have the option to accept or reject those changes.
+you have the option to accept or reject that change set.
 
 ```
 ubuntu:codebuild/ $ ./create_project.py --config codebuild-integv2.config --production --modify-existing
@@ -87,6 +89,12 @@ INFO:root:Update completed
 The [AWS Codebuild](https://docs.aws.amazon.com/codebuild/latest/userguide/limits.html) docs list the number of concurrent jobs at 60.
 With extensive testing, we've learned this number appears to be weighted based on [instance size](https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-compute-types.html) (or provisioning limits), so running all tests on the largest possible instances will reduce actual concurrency.  Additionally provisioning time is currently longer for larger instances, so there is a time penalty that might not be recovered by using a larger instance for short lived tests.
 
+### Batch Builds
+
+The `spec/buildspec_omnibus_batch.yml` contains a complete list of all CodeBuild jobs.  In the future, this will replace the individual jobs created by the create_project.py script.
+
+The broken out batch jobs: fuzz, integration and general, are created with the script create_batch.sh, which uses jq to parse out the jobs by title.
+
 ### Notes on moving from Travis-ci
 
 - Install_clang from Travis is using google chromium clang commit from 2017- which requires python2.7 (EOL); updated for CodeBuild.

codebuild/create_batch.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You may not use
+# this file except in compliance with the License. A copy of the License is
+# located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eu
+
+BUILDSPEC_OMNIBUS=./spec/buildspec_omnibus.yml
+BUILDSPEC_FUZZ=./spec/buildspec_fuzz_batch.yml
+BUILDSPEC_INTEG=./spec/buildspec_integ_batch.yml
+BUILDSPEC_GENERAL=./spec/buildspec_general_batch.yml
+
+synth_subjobs () {
+  yq -S -Y -r '{batch:{"build-list":[.batch."build-list"[]| select(.identifier|contains("Fuzz")) ]}}' $BUILDSPEC_OMNIBUS > $BUILDSPEC_FUZZ
+  yq -S -Y -r '{batch:{"build-list":[.batch."build-list"[]| select(.identifier|contains("Integ")) ]}}' $BUILDSPEC_OMNIBUS > $BUILDSPEC_INTEG
+  yq -S -Y -r '{batch:{"build-list":[.batch."build-list"[]| select(.identifier|contains("Fuzz")|not)|select(.identifier|contains("Integ")|not) ]}}' $BUILDSPEC_OMNIBUS > $BUILDSPEC_GENERAL
+}
+
+
+check_buildspec () {
+    OMNIBUS=$(yq -r '.batch."build-list"|length' $BUILDSPEC_OMNIBUS)
+    INTEG=$(yq -r '.batch."build-list"|length' $BUILDSPEC_INTEG)
+    FUZZ=$(yq -r '.batch."build-list"|length' $BUILDSPEC_FUZZ)
+    GENERAL=$(yq -r '.batch."build-list"|length' $BUILDSPEC_GENERAL)
+    echo -e "Checking newly created buildspec files\n$OMNIBUS = $INTEG + $FUZZ + $GENERAL"
+    if (($OMNIBUS != $INTEG+$FUZZ+$GENERAL)); then
+      echo "Counts do not match!"
+    fi
+}
+
+PREREQS="jq yq"
+for i in $PREREQS; do
+  if ! command -v $i &> /dev/null; then
+     echo "$i needs to be install (use pip)"
+  fi;
+done
+synth_subjobs
+check_buildspec
+echo "Note the buildspec_*_batch.yml files that were just created should only be used in-line with CodeBuild and not be commited to the repository."

codebuild/spec/buildspec_omnibus.yml

@@ -0,0 +1,239 @@
+---
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You may not use
+# this file except in compliance with the License. A copy of the License is
+# located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied. See the License for the specific language governing permissions and
+# limitations under the License.
+version: 0.2
+
+# Doc for batch https://docs.aws.amazon.com/codebuild/latest/userguide/batch-build-buildspec.html#build-spec.batch.build-list
+batch:
+  build-list:
+    # Consolidated Integration tests
+    - identifier: s2nIntegrationBoringLibre
+      buildspec: codebuild/spec/buildspec_ubuntu_integ_boringlibre.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+
+    - identifier: s2nIntegrationOpenSSL111PlusCoverage
+      buildspec: codebuild/spec/buildspec_ubuntu_integ_openssl111.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+
+    - identifier: s2nIntegrationOpenSSL102Plus
+      buildspec: codebuild/spec/buildspec_ubuntu_integ_openssl102.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+
+    - identifier: s2nIntegrationOpenSSL102AsanValgrind
+      buildspec: codebuild/spec/buildspec_ubuntu_integ_openssl102_asanvalgrind.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+
+    # Individual Integration tests
+    - identifier: s2nIntegrationBoringSSLGcc9
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: boringssl
+          BUILD_S2N: true
+          TESTS: integration
+          GCC_VERSION: 9
+
+    - identifier: s2nIntegrationOpenSSL111Gcc6SoftCrypto
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.1.1
+          BUILD_S2N: true
+          TESTS: integration
+          GCC_VERSION: 6
+          OPENSSL_ia32cap: "~0x200000200000000"
+
+    - identifier: s2nIntegrationOpenSSL111Gcc9
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.1.1
+          BUILD_S2N: true
+          TESTS: integration
+          GCC_VERSION: 9
+
+    - identifier: s2nIntegrationLibreSSLGcc9
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: libressl
+          BUILD_S2N: true
+          TESTS: integration
+          GCC_VERSION: 9
+
+    - identifier: s2nIntegrationOpenSSL111Gcc6Coverage
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.1.1
+          BUILD_S2N: true
+          TESTS: integration
+          GCC_VERSION: 6
+          S2N_COVERAGE: true
+          CODECOV_IO_UPLOAD: true
+
+    # Saw
+    - identifier: s2nSawBike
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_2XLARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.0.2
+          TESTS: sawBIKE
+          SAW: true
+          GCC_VERSION: NONE
+
+    - identifier: sawHMACPlus
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          TESTS: sawHMACPlus
+          SAW: true
+          GCC_VERSION: NONE
+
+    - identifier: s2nSawTls
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          TESTS: tls
+          SAW: true
+          GCC_VERSION: NONE
+
+    # Other
+    - identifier: s2nSidetrail
+      buildspec: codebuild/spec/buildspec_sidetrail.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_2XLARGE
+        image: 024603541914.dkr.ecr.us-west-2.amazonaws.com/linux-docker-images:sidetrail
+        variables:
+          TESTS: sidetrail
+
+
+    - identifier: s2nValgrindOpenSSL102Gcc6Fips
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          TESTS: valgrind
+          GCC_VERSION: 6
+          S2N_LIBCRYPTO: openssl-1.0.2-fips
+          BUILD_S2N: true
+
+    - identifier: s2nValgrindOpenSSL111Gcc9
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          TESTS: valgrind
+          GCC_VERSION: 9
+          S2N_LIBCRYPTO: openssl-1.1.1
+          BUILD_S2N: true
+
+    - identifier: s2nValgrindOpenssl102
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          TESTS: valgrind
+          GCC_VERSION: '6'
+          S2N_LIBCRYPTO: 'openssl-1.0.2'
+          BUILD_S2N: 'true'
+
+    - identifier: s2nAsanOpenSSL111Coverage
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_SMALL
+        variables:
+          TESTS: asan
+          GCC_VERSION: '6'
+          S2N_LIBCRYPTO: 'openssl-1.1.1'
+          BUILD_S2N: 'true'
+          S2N_COVERAGE: 'true'
+          CODECOV_IO_UPLOAD: 'true'
+          CODECOV_TOKEN: 'e460b7c1-6019-4a50-b65d-555c4a8fbc22'
+
+    - identifier: s2nAsanOpenssl102
+      buildspec: codebuild/spec/buildspec_ubuntu.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_SMALL
+        variables:
+          TESTS: asan
+          GCC_VERSION: '6'
+          S2N_LIBCRYPTO: 'openssl-1.0.2'
+          BUILD_S2N: 'true'
+
+    # Fuzz tests
+    - identifier: s2nFuzzerOpenSSL111Coverage
+      buildspec: codebuild/spec/buildspec_ubuntu_fuzz_artifacts.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.1.1
+          LATEST_CLANG: true
+          TESTS: fuzz
+          FUZZ_TIMEOUT_SEC: 60
+          FUZZ_COVERAGE: true
+          CODECOV_IO_UPLOAD: true
+
+    - identifier: s2nFuzzerOpenSSL102
+      buildspec: codebuild/spec/buildspec_ubuntu_fuzz_artifacts.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.0.2
+          LATEST_CLANG: true
+          TESTS: fuzz
+          FUZZ_TIMEOUT_SEC: 60
+
+    - identifier: s2nFuzzerOpenSSL102FIPS
+      buildspec: codebuild/spec/buildspec_ubuntu_fuzz_artifacts.yml
+      env:
+        privileged-mode: true
+        compute-type: BUILD_GENERAL1_LARGE
+        variables:
+          S2N_LIBCRYPTO: openssl-1.0.2-fips
+          LATEST_CLANG: true
+          TESTS: fuzz
+          FUZZ_TIMEOUT_SEC: 60
+