Skip to content

Commit 5837500

Browse files
hcho3hcho3
committed
[CI] Create a central place to update CI image tag
1 parent ea66308 commit 5837500

16 files changed

+51
-31
lines changed

‎doc/contrib/ci.rst

+17-9
Original file line numberDiff line numberDiff line change
@@ -43,18 +43,26 @@ To make changes to the CI container, carry out the following steps:
4343
Consult :ref:`build_run_docker_locally` for this step.
4444
4. Submit a pull request to `dmlc/xgboost-devops <https://github.com/dmlc/xgboost-devops>`_ with
4545
the proposed changes to the Dockerfile. Make note of the pull request number. Example: ``#204``
46-
5. Clone `dmlc/xgboost <https://github.com/dmlc/xgboost>`_ and update all references to the
47-
old container to point to the new container. More specifically, all container image URIs of form
48-
``492475357299.dkr.ecr.us-west-2.amazonaws.com/[image_repo]:main`` should have its image tag
49-
(last component) replaced with ``PR-#``, where ``#`` is the pull request number.
50-
For the example above,
51-
we'd replace ``492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:main`` with
52-
``492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu:PR-204``.
46+
5. Clone `dmlc/xgboost <https://github.com/dmlc/xgboost>`_. Locate the file
47+
``ops/pipeline/get-image-tag.sh``, which should have a single line
48+
49+
.. code-block:: bash
50+
51+
IMAGE_TAG=main
52+
53+
To use the new container, revise the file as follows:
54+
55+
.. code-block:: bash
56+
57+
IMAGE_TAG=PR-XX
58+
59+
where ``XX`` is the pull request number.
60+
5361
6. Now submit a pull request to `dmlc/xgboost <https://github.com/dmlc/xgboost>`_. The CI will
5462
run tests using the new container. Verify that all tests pass.
5563
7. Merge the pull request in ``dmlc/xgboost-devops``. Wait until the CI completes on the ``main`` branch.
56-
8. Go back to the the pull request for ``dmlc/xgboost`` and revise all the container references to use
57-
the old tag ``:main``.
64+
8. Go back to the the pull request for ``dmlc/xgboost`` and change ``ops/pipeline/get-image-tag.sh``
65+
back to ``IMAGE_TAG=main``.
5866
9. Merge the pull request in ``dmlc/xgboost``.
5967

6068
.. _build_run_docker_locally:

‎ops/pipeline/build-cpu-arm64.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,10 @@ fi
1111

1212
source ops/pipeline/classify-git-branch.sh
1313
source ops/pipeline/get-docker-registry-details.sh
14+
source ops/pipeline/get-image-tag.sh
1415

1516
WHEEL_TAG=manylinux_2_28_aarch64
16-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.aarch64:main
17+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.aarch64:${IMAGE_TAG}
1718

1819
echo "--- Build CPU code targeting ARM64"
1920
set -x

‎ops/pipeline/build-cpu.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,9 @@ set -euo pipefail
55

66
source ops/pipeline/classify-git-branch.sh
77
source ops/pipeline/get-docker-registry-details.sh
8+
source ops/pipeline/get-image-tag.sh
89

9-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.cpu:main
10+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.cpu:${IMAGE_TAG}
1011

1112
echo "--- Build CPU code"
1213
set -x

‎ops/pipeline/build-cuda.sh

+3-2
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,11 @@ esac
3333

3434
source ops/pipeline/classify-git-branch.sh
3535
source ops/pipeline/get-docker-registry-details.sh
36+
source ops/pipeline/get-image-tag.sh
3637

3738
WHEEL_TAG=manylinux_2_28_x86_64
38-
BUILD_IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:main"
39-
MANYLINUX_IMAGE_URI="${DOCKER_REGISTRY_URL}/xgb-ci.${WHEEL_TAG}:main"
39+
BUILD_IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:${IMAGE_TAG}"
40+
MANYLINUX_IMAGE_URI="${DOCKER_REGISTRY_URL}/xgb-ci.${WHEEL_TAG}:${IMAGE_TAG}"
4041

4142
echo "--- Build with CUDA"
4243

‎ops/pipeline/build-gpu-rpkg.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,9 @@ fi
1010

1111
source ops/pipeline/classify-git-branch.sh
1212
source ops/pipeline/get-docker-registry-details.sh
13+
source ops/pipeline/get-image-tag.sh
1314

14-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.gpu_build_r_rockylinux8:main
15+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.gpu_build_r_rockylinux8:${IMAGE_TAG}
1516

1617
echo "--- Build XGBoost R package with CUDA"
1718
set -x

‎ops/pipeline/build-jvm-doc.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,9 @@ then
1818
fi
1919

2020
source ops/pipeline/get-docker-registry-details.sh
21+
source ops/pipeline/get-image-tag.sh
2122

22-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm_gpu_build:main
23+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm_gpu_build:${IMAGE_TAG}
2324

2425
echo "--- Build JVM packages doc"
2526
set -x

‎ops/pipeline/build-jvm-gpu.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,9 @@ set -euo pipefail
55

66
source ops/pipeline/classify-git-branch.sh
77
source ops/pipeline/get-docker-registry-details.sh
8+
source ops/pipeline/get-image-tag.sh
89

9-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm_gpu_build:main
10+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm_gpu_build:${IMAGE_TAG}
1011

1112
echo "--- Build libxgboost4j.so with CUDA"
1213

‎ops/pipeline/build-jvm-manylinux2014.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,9 @@ image_repo="xgb-ci.manylinux2014_${arch}"
1414

1515
source ops/pipeline/classify-git-branch.sh
1616
source ops/pipeline/get-docker-registry-details.sh
17+
source ops/pipeline/get-image-tag.sh
1718

18-
IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:main"
19+
IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:${IMAGE_TAG}"
1920

2021
# Build XGBoost4J binary
2122
echo "--- Build libxgboost4j.so (targeting glibc 2.17)"

‎ops/pipeline/build-manylinux2014.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,11 @@ arch="$1"
1818

1919
source ops/pipeline/classify-git-branch.sh
2020
source ops/pipeline/get-docker-registry-details.sh
21+
source ops/pipeline/get-image-tag.sh
2122

2223
WHEEL_TAG="manylinux2014_${arch}"
2324
IMAGE_REPO="xgb-ci.${WHEEL_TAG}"
24-
IMAGE_URI="${DOCKER_REGISTRY_URL}/${IMAGE_REPO}:main"
25+
IMAGE_URI="${DOCKER_REGISTRY_URL}/${IMAGE_REPO}:${IMAGE_TAG}"
2526
PYTHON_BIN="/opt/python/cp310-cp310/bin/python"
2627

2728
echo "--- Build binary wheel for ${WHEEL_TAG}"

‎ops/pipeline/build-test-jvm-packages.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ EOF
1313
set -euo pipefail
1414

1515
source ops/pipeline/get-docker-registry-details.sh
16+
source ops/pipeline/get-image-tag.sh
1617

1718
for arg in "SCALA_VERSION"
1819
do
@@ -23,7 +24,7 @@ do
2324
fi
2425
done
2526

26-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm:main
27+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm:${IMAGE_TAG}
2728

2829
set -x
2930

‎ops/pipeline/deploy-jvm-packages.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ set -euo pipefail
55

66
source ops/pipeline/enforce-ci.sh
77
source ops/pipeline/get-docker-registry-details.sh
8+
source ops/pipeline/get-image-tag.sh
89

910
if [[ "$#" -lt 3 ]]
1011
then
@@ -16,7 +17,7 @@ variant="$1"
1617
image_repo="$2"
1718
scala_version="$3"
1819

19-
IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:main"
20+
IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:${IMAGE_TAG}"
2021

2122
set -x
2223

‎ops/pipeline/get-image-tag.sh

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
## Update the following line to test changes to CI images
2+
## See https://xgboost.readthedocs.io/en/latest/contrib/ci.html#making-changes-to-ci-containers
3+
4+
IMAGE_TAG=PR-11

‎ops/pipeline/run-clang-tidy.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@
33
set -euo pipefail
44

55
source ops/pipeline/get-docker-registry-details.sh
6+
source ops/pipeline/get-image-tag.sh
67

7-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.clang_tidy:main
8+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.clang_tidy:${IMAGE_TAG}
89

910
echo "--- Run clang-tidy"
1011
set -x

‎ops/pipeline/test-cpp-i386.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@
44
set -euo pipefail
55

66
source ops/pipeline/get-docker-registry-details.sh
7+
source ops/pipeline/get-image-tag.sh
78

8-
IMAGE_URI="${DOCKER_REGISTRY_URL}/xgb-ci.i386:main"
9+
IMAGE_URI="${DOCKER_REGISTRY_URL}/xgb-ci.i386:${IMAGE_TAG}"
910

1011
set -x
1112
python3 ops/docker_run.py \

‎ops/pipeline/test-jvm-gpu.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,9 @@ do
2424
done
2525

2626
source ops/pipeline/get-docker-registry-details.sh
27+
source ops/pipeline/get-image-tag.sh
2728

28-
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm_gpu_build:main
29+
IMAGE_URI=${DOCKER_REGISTRY_URL}/xgb-ci.jvm_gpu_build:${IMAGE_TAG}
2930

3031
set -x
3132

‎ops/pipeline/test-python-wheel.sh

+3-8
Original file line numberDiff line numberDiff line change
@@ -19,15 +19,10 @@ else
1919
gpu_option=""
2020
fi
2121

22-
if [[ "$image_repo" == "xgb-ci.gpu" ]]
23-
then
24-
image_tag="PR-11"
25-
else
26-
image_tag="main"
27-
fi
28-
2922
source ops/pipeline/get-docker-registry-details.sh
30-
IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:${image_tag}"
23+
source ops/pipeline/get-image-tag.sh
24+
25+
IMAGE_URI="${DOCKER_REGISTRY_URL}/${image_repo}:${IMAGE_TAG}"
3126

3227
set -x
3328
python3 ops/docker_run.py --image-uri "${IMAGE_URI}" ${gpu_option} \

0 commit comments

Comments
 (0)