Skip to content

Commit 371f587

Browse files
huydhnpytorchmergebot
huydhn
authored and
pytorchmergebot
committed
Dockerize lint jobs (pytorch#94255)
This is to minimize network flakiness when running lint jobs. I create a new Docker image for linter and install all linter dependencies there. After that, all linter jobs are converted to use Nova generic Linux job https://github.com/pytorch/test-infra/blob/main/.github/workflows/linux_job.yml with the new image. For the future task: I encounter this issue with the current mypy version we are using and Python 3.11 python/mypy#13627. Fixing this requires upgrading mypy to a newer version, but that can be done separately (require formatting/fixing `*.py` files with the newer mypy version) `collect_env` linter job is currently not included here as it needs older Python versions (3.5). It could also be converted to use the same mechanism (with another Docker image, probably). This one rarely fails though. ### Testing BEFORE https://github.com/pytorch/pytorch/actions/runs/4130366955 took a total of ~14m AFTER https://github.com/pytorch/pytorch/actions/runs/4130712385 also takes a total of ~14m Pull Request resolved: pytorch#94255 Approved by: https://github.com/ZainRizvi
1 parent abfd293 commit 371f587

File tree

15 files changed

+324
-284
lines changed

15 files changed

+324
-284
lines changed

‎.ci/docker/build.sh

+10
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,9 @@ if [[ "$image" == *cuda* && "$UBUNTU_VERSION" != "22.04" ]]; then
7373
DOCKERFILE="${OS}-cuda/Dockerfile"
7474
elif [[ "$image" == *rocm* ]]; then
7575
DOCKERFILE="${OS}-rocm/Dockerfile"
76+
elif [[ "$image" == *linter* ]]; then
77+
# Use a separate Dockerfile for linter to keep a small image size
78+
DOCKERFILE="linter/Dockerfile"
7679
fi
7780

7881
# CMake 3.18 is needed to support CUDA17 language variant
@@ -234,6 +237,13 @@ case "$image" in
234237
DB=yes
235238
VISION=yes
236239
;;
240+
pytorch-linux-focal-linter)
241+
# TODO: Use 3.9 here because of this issue https://github.com/python/mypy/issues/13627.
242+
# We will need to update mypy version eventually, but that's for another day. The task
243+
# would be to upgrade mypy to 1.0.0 with Python 3.11
244+
ANACONDA_PYTHON_VERSION=3.9
245+
CONDA_CMAKE=yes
246+
;;
237247
*)
238248
# Catch-all for builds that are not hardcoded.
239249
PROTOBUF=yes

‎.ci/docker/centos-rocm/Dockerfile

+2-2
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,8 @@ ENV ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION
4545
ENV PATH /opt/conda/envs/py_$ANACONDA_PYTHON_VERSION/bin:/opt/conda/bin:$PATH
4646
COPY requirements-ci.txt /opt/conda/requirements-ci.txt
4747
COPY ./common/install_conda.sh install_conda.sh
48-
RUN bash ./install_conda.sh && rm install_conda.sh
49-
RUN rm /opt/conda/requirements-ci.txt
48+
COPY ./common/common_utils.sh common_utils.sh
49+
RUN bash ./install_conda.sh && rm install_conda.sh common_utils.sh /opt/conda/requirements-ci.txt
5050

5151
# (optional) Install protobuf for ONNX
5252
ARG PROTOBUF

‎.ci/docker/common/common_utils.sh

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/bin/bash
2+
3+
# Work around bug where devtoolset replaces sudo and breaks it.
4+
if [ -n "$DEVTOOLSET_VERSION" ]; then
5+
export SUDO=/bin/sudo
6+
else
7+
export SUDO=sudo
8+
fi
9+
10+
as_jenkins() {
11+
# NB: unsetting the environment variables works around a conda bug
12+
# https://github.com/conda/conda/issues/6576
13+
# NB: Pass on PATH and LD_LIBRARY_PATH to sudo invocation
14+
# NB: This must be run from a directory that jenkins has access to,
15+
# works around https://github.com/conda/conda-package-handling/pull/34
16+
$SUDO -H -u jenkins env -u SUDO_UID -u SUDO_GID -u SUDO_COMMAND -u SUDO_USER env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
17+
}
18+
19+
conda_install() {
20+
# Ensure that the install command don't upgrade/downgrade Python
21+
# This should be called as
22+
# conda_install pkg1 pkg2 ... [-c channel]
23+
as_jenkins conda install -q -n py_$ANACONDA_PYTHON_VERSION -y python="$ANACONDA_PYTHON_VERSION" $*
24+
}
25+
26+
conda_run() {
27+
as_jenkins conda run -n py_$ANACONDA_PYTHON_VERSION --no-capture-output $*
28+
}
29+
30+
pip_install() {
31+
as_jenkins conda run -n py_$ANACONDA_PYTHON_VERSION pip install --progress-bar off $*
32+
}

‎.ci/docker/common/install_conda.sh

+1-26
Original file line numberDiff line numberDiff line change
@@ -24,21 +24,7 @@ if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
2424
mkdir -p /opt/conda
2525
chown jenkins:jenkins /opt/conda
2626

27-
# Work around bug where devtoolset replaces sudo and breaks it.
28-
if [ -n "$DEVTOOLSET_VERSION" ]; then
29-
SUDO=/bin/sudo
30-
else
31-
SUDO=sudo
32-
fi
33-
34-
as_jenkins() {
35-
# NB: unsetting the environment variables works around a conda bug
36-
# https://github.com/conda/conda/issues/6576
37-
# NB: Pass on PATH and LD_LIBRARY_PATH to sudo invocation
38-
# NB: This must be run from a directory that jenkins has access to,
39-
# works around https://github.com/conda/conda-package-handling/pull/34
40-
$SUDO -H -u jenkins env -u SUDO_UID -u SUDO_GID -u SUDO_COMMAND -u SUDO_USER env "PATH=$PATH" "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" $*
41-
}
27+
source "$(dirname "${BASH_SOURCE[0]}")/common_utils.sh"
4228

4329
pushd /tmp
4430
wget -q "${BASE_URL}/${CONDA_FILE}"
@@ -63,17 +49,6 @@ if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
6349
# Install correct Python version
6450
as_jenkins conda create -n py_$ANACONDA_PYTHON_VERSION -y python="$ANACONDA_PYTHON_VERSION"
6551

66-
conda_install() {
67-
# Ensure that the install command don't upgrade/downgrade Python
68-
# This should be called as
69-
# conda_install pkg1 pkg2 ... [-c channel]
70-
as_jenkins conda install -q -n py_$ANACONDA_PYTHON_VERSION -y python="$ANACONDA_PYTHON_VERSION" $*
71-
}
72-
73-
pip_install() {
74-
as_jenkins conda run -n py_$ANACONDA_PYTHON_VERSION pip install --progress-bar off $*
75-
}
76-
7752
# Install PyTorch conda deps, as per https://github.com/pytorch/pytorch README
7853
CONDA_COMMON_DEPS="astunparse pyyaml mkl=2021.4.0 mkl-include=2021.4.0 setuptools"
7954
if [ "$ANACONDA_PYTHON_VERSION" = "3.11" ]; then

‎.ci/docker/common/install_linter.sh

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/bin/bash
2+
3+
set -ex
4+
5+
source "$(dirname "${BASH_SOURCE[0]}")/common_utils.sh"
6+
7+
if [ -n "${UBUNTU_VERSION}" ]; then
8+
apt update
9+
apt-get install -y clang doxygen git graphviz nodejs npm libtinfo5
10+
fi
11+
12+
# Do shallow clone of PyTorch so that we can init lintrunner in Docker build context
13+
git clone https://github.com/pytorch/pytorch.git --depth 1
14+
chown -R jenkins pytorch
15+
16+
pushd pytorch
17+
# Install all linter dependencies
18+
pip_install -r requirements.txt
19+
conda_run lintrunner init
20+
popd
21+
22+
# Node dependencies required by toc linter job
23+
npm install -g markdown-toc
24+
25+
# Cleaning up
26+
rm -rf pytorch

‎.ci/docker/linter/Dockerfile

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
ARG UBUNTU_VERSION
2+
3+
FROM ubuntu:${UBUNTU_VERSION}
4+
5+
ARG UBUNTU_VERSION
6+
7+
ENV DEBIAN_FRONTEND noninteractive
8+
9+
# Install common dependencies (so that this step can be cached separately)
10+
COPY ./common/install_base.sh install_base.sh
11+
RUN bash ./install_base.sh && rm install_base.sh
12+
13+
# Install user
14+
COPY ./common/install_user.sh install_user.sh
15+
RUN bash ./install_user.sh && rm install_user.sh
16+
17+
# Install conda and other packages (e.g., numpy, pytest)
18+
ARG ANACONDA_PYTHON_VERSION
19+
ARG CONDA_CMAKE
20+
ENV ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION
21+
ENV PATH /opt/conda/envs/py_$ANACONDA_PYTHON_VERSION/bin:/opt/conda/bin:$PATH
22+
COPY requirements-ci.txt /opt/conda/requirements-ci.txt
23+
COPY ./common/install_conda.sh install_conda.sh
24+
COPY ./common/common_utils.sh common_utils.sh
25+
RUN bash ./install_conda.sh && rm install_conda.sh common_utils.sh /opt/conda/requirements-ci.txt
26+
27+
# Note that Docker build forbids copying file outside the build context
28+
COPY ./common/install_linter.sh install_linter.sh
29+
COPY ./common/common_utils.sh common_utils.sh
30+
RUN bash ./install_linter.sh
31+
RUN rm install_linter.sh common_utils.sh
32+
33+
USER jenkins
34+
CMD ["bash"]

‎.ci/docker/requirements-ci.txt

+15
Original file line numberDiff line numberDiff line change
@@ -248,3 +248,18 @@ unittest-xml-reporting<=3.2.0,>=2.0.0
248248
#Description: saves unit test results to xml
249249
#Pinned versions:
250250
#test that import:
251+
252+
lintrunner==0.9.2
253+
#Description: all about linters
254+
#Pinned versions: 0.9.2
255+
#test that import:
256+
257+
rockset==1.0.3
258+
#Description: queries Rockset
259+
#Pinned versions: 1.0.3
260+
#test that import:
261+
262+
ghstack==0.7.1
263+
#Description: ghstack tool
264+
#Pinned versions: 0.7.1
265+
#test that import:

‎.ci/docker/ubuntu-cuda/Dockerfile

+2-2
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ ENV PATH /opt/conda/envs/py_$ANACONDA_PYTHON_VERSION/bin:/opt/conda/bin:$PATH
2929
ARG CONDA_CMAKE
3030
COPY requirements-ci.txt /opt/conda/requirements-ci.txt
3131
COPY ./common/install_conda.sh install_conda.sh
32-
RUN bash ./install_conda.sh && rm install_conda.sh
33-
RUN rm /opt/conda/requirements-ci.txt
32+
COPY ./common/common_utils.sh common_utils.sh
33+
RUN bash ./install_conda.sh && rm install_conda.sh common_utils.sh /opt/conda/requirements-ci.txt
3434

3535
# Install gcc
3636
ARG GCC_VERSION

‎.ci/docker/ubuntu-rocm/Dockerfile

+2-2
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ ENV ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION
3131
ENV PATH /opt/conda/envs/py_$ANACONDA_PYTHON_VERSION/bin:/opt/conda/bin:$PATH
3232
COPY requirements-ci.txt /opt/conda/requirements-ci.txt
3333
COPY ./common/install_conda.sh install_conda.sh
34-
RUN bash ./install_conda.sh && rm install_conda.sh
35-
RUN rm /opt/conda/requirements-ci.txt
34+
COPY ./common/common_utils.sh common_utils.sh
35+
RUN bash ./install_conda.sh && rm install_conda.sh common_utils.sh /opt/conda/requirements-ci.txt
3636

3737
# Install gcc
3838
ARG GCC_VERSION

‎.ci/docker/ubuntu/Dockerfile

+2-6
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ ENV ANACONDA_PYTHON_VERSION=$ANACONDA_PYTHON_VERSION
4040
ENV PATH /opt/conda/envs/py_$ANACONDA_PYTHON_VERSION/bin:/opt/conda/bin:$PATH
4141
COPY requirements-ci.txt /opt/conda/requirements-ci.txt
4242
COPY ./common/install_conda.sh install_conda.sh
43-
RUN bash ./install_conda.sh && rm install_conda.sh
44-
RUN rm /opt/conda/requirements-ci.txt
43+
COPY ./common/common_utils.sh common_utils.sh
44+
RUN bash ./install_conda.sh && rm install_conda.sh common_utils.sh /opt/conda/requirements-ci.txt
4545

4646
# Install gcc
4747
ARG GCC_VERSION
@@ -137,10 +137,6 @@ RUN rm install_openssl.sh
137137
# Install ccache/sccache (do this last, so we get priority in PATH)
138138
COPY ./common/install_cache.sh install_cache.sh
139139
ENV PATH /opt/cache/bin:$PATH
140-
# See https://github.com/pytorch/pytorch/issues/82174
141-
# TODO([email protected]):
142-
# check if this is needed after full off Xenial migration
143-
ENV CARGO_NET_GIT_FETCH_WITH_CLI true
144140
RUN bash ./install_cache.sh && rm install_cache.sh
145141

146142
# Add jni.h for java host build

‎.github/workflows/_calculate-docker-image.yml

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: calculate-docker-image
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
docker-image-name:
7+
required: true
8+
type: string
9+
description: Name of the base docker image to build with.
10+
11+
outputs:
12+
docker-image:
13+
value: ${{ jobs.calculate-docker-image.outputs.docker-image }}
14+
description: The docker image containing the built PyTorch.
15+
16+
jobs:
17+
calculate-docker-image:
18+
if: github.repository_owner == 'pytorch'
19+
runs-on: [self-hosted, linux.large]
20+
timeout-minutes: 15
21+
outputs:
22+
docker-image: ${{ steps.calculate-docker-image.outputs.docker-image }}
23+
steps:
24+
- name: Checkout PyTorch
25+
uses: pytorch/pytorch/.github/actions/checkout-pytorch@master
26+
with:
27+
submodules: false
28+
fetch-depth: 1
29+
30+
- name: Setup Linux
31+
uses: ./.github/actions/setup-linux
32+
33+
- name: Calculate docker image
34+
id: calculate-docker-image
35+
uses: ./.github/actions/calculate-docker-image
36+
with:
37+
docker-image-name: ${{ inputs.docker-image-name }}

‎.github/workflows/docker-builds.yml

+1
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ jobs:
4848
- docker-image-name: pytorch-linux-focal-py3.8-gcc7
4949
- docker-image-name: pytorch-linux-focal-py3-clang7-asan
5050
- docker-image-name: pytorch-linux-focal-py3-clang10-onnx
51+
- docker-image-name: pytorch-linux-focal-linter
5152
env:
5253
DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/${{ matrix.docker-image-name }}
5354
steps:

0 commit comments

Comments
 (0)