Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-15963: add r tweedie package to our test images #15964

Merged
merged 50 commits into from
Feb 20, 2024
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
af4b9fd
Add tweedie package
tomasfryda Dec 11, 2023
8e3861f
fix pyarrow typo
tomasfryda Dec 11, 2023
87ba69e
Increase the DEFAULT_IMAGE_VERSION_TAG in scripts/jenkins/groovy/buil…
tomasfryda Dec 11, 2023
55c9c01
Add missing quotes to test-requirements.txt
tomasfryda Dec 11, 2023
0de141a
Increase DEFAULT_IMAGE_VERSION_TAG
tomasfryda Dec 12, 2023
e578cf7
Fix more typos
tomasfryda Dec 12, 2023
615c3d5
specify python versions for packages we had issues in the build process
tomasfryda Dec 12, 2023
773dcdc
compile python3.6
tomasfryda Dec 13, 2023
c47915d
Add htmlTable for Hmisc in R
tomasfryda Dec 13, 2023
8a5b410
Install old pkgs
tomasfryda Dec 15, 2023
56b5e4b
Update Dockerfile-build-base
tomasfryda Dec 18, 2023
29aca57
fix issues with nodesource setup script in python and jdk-other images
tomasfryda Jan 2, 2024
3de9fa3
Extract NODE_MAJOR env var ouside of the complicated RUN statements
tomasfryda Jan 2, 2024
269f417
Add --no-tty to gpg call to resolve issue with non-existing /dev/tty
tomasfryda Jan 3, 2024
f1dcf2d
Enable Tweedie test
tomasfryda Jan 4, 2024
3f2e786
Use make install instead of make altinstall when installing python 3.…
tomasfryda Jan 5, 2024
e0c36cc
Increase default build python version to 3.7
tomasfryda Jan 8, 2024
a7870d7
Update R library path due to new OS version
tomasfryda Jan 8, 2024
29a38c1
Install chromium directly (apt now installs just a file that tells yo…
tomasfryda Jan 10, 2024
a38f6db
Update python 3.6 pkgs to resolve statsmodels and scipy version incom…
tomasfryda Jan 12, 2024
373556d
update R tests
tomasfryda Jan 16, 2024
fcd2cad
Add new error message for cacert test
tomasfryda Jan 16, 2024
51d8f24
Deal with lifelines >= 0.27 incompatibility with py < 3.9
tomasfryda Jan 16, 2024
58aab3c
Add more debugging info to the Tweedie test
tomasfryda Jan 16, 2024
ee38e5f
Fix runit_pubdev_8683_glm_gamma_dispersion_factor_large.R
tomasfryda Jan 16, 2024
619bce2
fix matplotlib issue in python demos and add more debugging info to t…
tomasfryda Jan 17, 2024
8bfb7da
Fix h2o-py/demos/uplift_random_forest_compare_causalml.ipynb by remov…
tomasfryda Jan 17, 2024
20a3d2f
Make use of datatable and polars opt-in not opt-out
tomasfryda Jan 18, 2024
5c8d97e
Make r tweedie dispersion test more reproducible
tomasfryda Jan 18, 2024
5594e32
Fix logical error in an exceptional case in Tweedie dispersion estima…
tomasfryda Jan 19, 2024
9de4cf3
log more information about Tweedie dispersion estimation
tomasfryda Jan 19, 2024
4711d1b
Increase tolerance in pyunit_PUBDEV_6117_xgboost_compare.py
tomasfryda Jan 19, 2024
7f94689
Disable Newton's methon in dispersion estimation in GLM Tweedie
tomasfryda Jan 21, 2024
518038a
Fix pubdev_8685_tweedie_dispersion_factor_exceed2
tomasfryda Jan 22, 2024
09dfcd0
Increase tolerance in pyunit_PUBDEV_6117_xgboost_compare.py to 1e-6
tomasfryda Jan 23, 2024
689520a
Fix models.R comments (roxygen complains)
tomasfryda Jan 23, 2024
6d3b9f5
Fix tweedie dispersion estimation for variance power >= 2 with respon…
tomasfryda Jan 24, 2024
72be72d
Fix long line in R comment for CRAN checks
tomasfryda Jan 24, 2024
6d69014
Move model_pearson back where it used to be in tweedie dispersion fac…
tomasfryda Jan 25, 2024
c28ef0b
Remove an unnecessary space
tomasfryda Jan 25, 2024
3dc2316
Fix R3.5.3 init java 11 test
tomasfryda Jan 29, 2024
0f97d8a
Make runit_pubdev_8683_glm_gamma_dispersion_factor_large.R more robus…
tomasfryda Jan 31, 2024
b68c59b
Remove packages for python < 3.5 from test-requirements.txt
tomasfryda Feb 1, 2024
73bb536
Try to fix R issues when building on jenkins
tomasfryda Feb 5, 2024
2c36fd3
Fix 'The process cannot access the file because it is being used by a…
tomasfryda Feb 5, 2024
80a6a23
Make sure R images contain all the suggested packages
tomasfryda Feb 7, 2024
00dc108
Update roxygen and rlang in R images
tomasfryda Feb 8, 2024
b9bf5ae
Update pkgdown
tomasfryda Feb 14, 2024
c6ffe57
Add mypy and specify the explicit version of already present typing_e…
tomasfryda Feb 14, 2024
50ad27b
Increase timeout in stages that are too often aborted
tomasfryda Feb 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 15 additions & 11 deletions docker/jenkins-images/Dockerfile-build-base
Original file line number Diff line number Diff line change
@@ -1,17 +1,21 @@
ARG FROM_VERSION
FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-r-3.4.1:${FROM_VERSION}

RUN \
curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way of installing Nodejs is deprecated and the script will get removed. So I updated the way to install nodejs==16.

apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
texlive \
texlive-fonts-extra \
texlive-htmlxml \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

texinfo \
texlive-bibtex-extra \
texlive-formats-extra \
texlive-generic-extra && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

texlive-generic-extra doesn't exist anymore. It used to be a transitional package that installed texlive-plain-generic which I added in the previous step when removing texlive-htmlxml.

See https://answers.launchpad.net/ubuntu/bionic/amd64/texlive-generic-extra/2017.20180305-1

ENV NODE_MAJOR '16'

RUN apt-get update && apt-get install -y ca-certificates curl gnupg && mkdir -p /etc/apt/keyrings && \
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --no-tty --yes --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
nodejs \
texlive \
texlive-fonts-extra \
texlive-latex-extra \
texlive-latex-recommended \
texlive-plain-generic \
texinfo \
texlive-bibtex-extra \
texlive-formats-extra && \
apt-get clean && \
apt-get autoremove -y && \
rm -rf /var/cache/apt/*
Expand Down
6 changes: 5 additions & 1 deletion docker/jenkins-images/Dockerfile-jdk-others-base
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
ARG FROM_VERSION
FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-base:${FROM_VERSION}

ENV NODE_MAJOR '16'

RUN \
add-apt-repository -y ppa:deadsnakes && \
curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script is deprecated and doesn't work anymore

apt-get update && apt-get install -y ca-certificates curl gnupg && mkdir -p /etc/apt/keyrings && \
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --no-tty --yes --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
python3-virtualenv libkrb5-dev nodejs && \
Expand Down
16 changes: 11 additions & 5 deletions docker/jenkins-images/Dockerfile-python-base
Original file line number Diff line number Diff line change
@@ -1,21 +1,27 @@
ARG FROM_VERSION
FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-jdk-8-base:${FROM_VERSION}

ENV NODE_MAJOR '16'

RUN \
add-apt-repository -y ppa:deadsnakes && \
curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
apt-get update && apt-get install -y ca-certificates curl gnupg && mkdir -p /etc/apt/keyrings && \
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --no-tty --yes --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
apt-get update -q -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 \
libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 \
libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libx11-xcb-dev libxcb1 libxcomposite1 libxcomposite-dev \
libxcursor1 libxcursor-dev libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils libblas-dev liblapack-dev \
chromium-browser \
libkrb5-dev \
nodejs && \
libgbm1 libkrb5-dev nodejs && \
apt-get clean && \
rm -rf /var/cache/apt/*
rm -rf /var/cache/apt/* && \
wget "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Linux_x64%2F1244635%2Fchrome-linux.zip?alt=media" -O chrome-linux.zip && \
unzip chrome-linux.zip -d /opt/ && \
rm chrome-linux.zip && \
ln -s /opt/chrome-linux/chrome /usr/bin/chromium-browser

ENV CHROME_BIN=/usr/bin/chromium-browser

Expand Down
10 changes: 10 additions & 0 deletions docker/jenkins-images/Dockerfile-r
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-r-base:${FROM_VERSION}
ARG R_VERSION
ENV R_VERSION=${R_VERSION}

# Old R versions (at least 3.3 and 3.4) are written in a way that is not compilable. This is likely due to the change in
# gcc-10 (GCC now defaults to -fno-common. As a result, global variable accesses are more efficient on various targets.
# In C, global variables with multiple tentative definitions now result in linker errors. With -fcommon such definitions
# are silently merged during linking.). I tried using -fcommon but without much luck so due to the time constrain I
# decided to use gcc-9 instead.
RUN apt-get update && apt-get install -y gcc-9 gfortran-9 g++-9 tcl-dev tk-dev && apt-get clean && apt-get autoremove -y && \
rm -rf /var/cache/apt/* &&\
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 99 --slave /usr/bin/g++ g++ /usr/bin/g++-9 \
--slave /usr/bin/gcov gcov /usr/bin/gcov-9 && update-alternatives --install /usr/bin/gfortran f95 /usr/bin/gfortran-9 99

COPY scripts/install_R_version /tmp/
RUN \
chmod +x /tmp/install_R_version && \
Expand Down
2 changes: 1 addition & 1 deletion docker/scripts/build-h2o-3
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ if [[ ! ${H2O_BRANCH} ]]; then
fi
if [[ ! -n ${PYTHON_VERSION} ]]; then
echo "Using default Python version"
PYTHON_VERSION='3.6'
PYTHON_VERSION='3.7'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python 3.6 is not supported anymore by the ubuntu 22.04 and is installed (built from source) separately for the py-3.6 tests.

fi

# cd home
Expand Down
Loading
Loading