GH-15963: add r tweedie package to our test images (#15964)

* Add tweedie package

* fix pyarrow typo

* Increase the DEFAULT_IMAGE_VERSION_TAG in scripts/jenkins/groovy/buildConfig.groovy

* Add missing quotes to test-requirements.txt

* Increase DEFAULT_IMAGE_VERSION_TAG

* Fix more typos

* specify python versions for packages we had issues in the build process

* compile python3.6

Changed version of wheel as 0.38.0 was yanked out of pypi due to
circular dependency

* Add htmlTable for Hmisc in R

* Install old pkgs

* Update  Dockerfile-build-base

* fix issues with nodesource setup script in python and jdk-other images

* Extract NODE_MAJOR env var ouside of the complicated RUN statements

* Add --no-tty to gpg call to resolve issue with non-existing /dev/tty

* Enable Tweedie test

* Use make install instead of make altinstall when installing python 3.6 to solve issue with virtualenv

* Increase default build python version to 3.7

* Update R library path due to new OS version

* Install chromium directly (apt now installs just a file that tells you to install it using snap which requires running systemd in the docker)

* Update python 3.6 pkgs to resolve statsmodels and scipy version incompatibility

* update R tests

* Add new error message for cacert test

* Deal with lifelines >= 0.27 incompatibility with py < 3.9

* Add more debugging info to the Tweedie test

* Fix runit_pubdev_8683_glm_gamma_dispersion_factor_large.R

* fix matplotlib issue in python demos and add more debugging info to tweedie ml dispersion estimation

* Fix h2o-py/demos/uplift_random_forest_compare_causalml.ipynb by removing the treatment from test set during prediction

* Make use of datatable and polars opt-in not opt-out

* Make r tweedie dispersion test more reproducible

* Fix logical error in an exceptional case in Tweedie dispersion estimation

* log more information about Tweedie dispersion estimation

* Increase tolerance in pyunit_PUBDEV_6117_xgboost_compare.py

* Disable Newton's methon in dispersion estimation in GLM Tweedie

* Fix pubdev_8685_tweedie_dispersion_factor_exceed2

* Increase tolerance in pyunit_PUBDEV_6117_xgboost_compare.py to 1e-6

* Fix models.R comments (roxygen complains)

* Fix tweedie dispersion estimation for variance power >= 2 with response == 0

* Fix long line in R comment for CRAN checks

* Move model_pearson back where it used to be in tweedie dispersion factor test

* Remove an unnecessary space

* Fix R3.5.3 init java 11 test

* Make runit_pubdev_8683_glm_gamma_dispersion_factor_large.R more robust - if R GLM fails to converge don't fail the test

* Remove packages for python  < 3.5 from test-requirements.txt

* Try to fix R issues when building on jenkins

Basically, some files are not used in the final package probably due to
some non-deterministic issue in roxygen2 which fills our NAMESPACE file
and creates our R man pages.

* Fix 'The process cannot access the file because it is being used by another process' on windows in python

* Make sure R images contain all the suggested packages

* Update roxygen and rlang in R images

* Update pkgdown

* Add mypy and specify the explicit version of already present typing_extensions

* Increase timeout in stages that are too often aborted

The R medium stages didn't run tests that contained missing package so
it's understandable that the runtime there increased. Java JUnit tests
also are aborted very often in master so this PR likely doesn't cause
the issue.

docker/jenkins-images/Dockerfile-build-base

@@ -1,17 +1,21 @@
 ARG FROM_VERSION
 FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-r-3.4.1:${FROM_VERSION}
 
-RUN \
-    curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
-    apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y \
-        texlive \
-        texlive-fonts-extra \
-        texlive-htmlxml \
-        texinfo \
-        texlive-bibtex-extra \
-        texlive-formats-extra \
-        texlive-generic-extra && \
+ENV NODE_MAJOR '16'
+
+RUN apt-get update && apt-get install -y ca-certificates curl gnupg &&  mkdir -p /etc/apt/keyrings && \
+curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --no-tty --yes --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
+echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
+DEBIAN_FRONTEND=noninteractive apt-get install -y \
+    nodejs \
+    texlive \
+    texlive-fonts-extra \
+    texlive-latex-extra \
+    texlive-latex-recommended \
+    texlive-plain-generic \
+    texinfo \
+    texlive-bibtex-extra \
+    texlive-formats-extra && \
     apt-get clean && \
     apt-get autoremove -y && \
     rm -rf /var/cache/apt/*

docker/jenkins-images/Dockerfile-jdk-others-base

@@ -1,9 +1,13 @@
 ARG FROM_VERSION
 FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-base:${FROM_VERSION}
 
+ENV NODE_MAJOR '16'
+
 RUN \
     add-apt-repository -y ppa:deadsnakes && \
-    curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
+    apt-get update && apt-get install -y ca-certificates curl gnupg &&  mkdir -p /etc/apt/keyrings && \
+    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --no-tty --yes --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
+    echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
     apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt-get install -y \
         python3-virtualenv libkrb5-dev nodejs && \

docker/jenkins-images/Dockerfile-python-base

@@ -1,21 +1,27 @@
 ARG FROM_VERSION
 FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-jdk-8-base:${FROM_VERSION}
 
+ENV NODE_MAJOR '16'
+
 RUN \
   add-apt-repository -y ppa:deadsnakes && \
-  curl -sL https://deb.nodesource.com/setup_16.x | bash - && \
+  apt-get update && apt-get install -y ca-certificates curl gnupg &&  mkdir -p /etc/apt/keyrings && \
+  curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --no-tty --yes --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
+  echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
   apt-get update -q -y && \
   DEBIAN_FRONTEND=noninteractive apt-get install -y \
         gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 \
         libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 \
         libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libx11-xcb-dev libxcb1 libxcomposite1 libxcomposite-dev \
         libxcursor1 libxcursor-dev libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
         ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils libblas-dev liblapack-dev \
-        chromium-browser \
-        libkrb5-dev \
-        nodejs && \
+        libgbm1 libkrb5-dev nodejs && \
     apt-get clean && \
-    rm -rf /var/cache/apt/*
+    rm -rf /var/cache/apt/* && \
+    wget "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Linux_x64%2F1244635%2Fchrome-linux.zip?alt=media" -O chrome-linux.zip && \
+    unzip chrome-linux.zip -d /opt/ && \
+    rm chrome-linux.zip && \
+    ln -s /opt/chrome-linux/chrome /usr/bin/chromium-browser
 
 ENV CHROME_BIN=/usr/bin/chromium-browser
 

docker/jenkins-images/Dockerfile-r

@@ -4,6 +4,16 @@ FROM harbor.h2o.ai/opsh2oai/h2o-3/dev-r-base:${FROM_VERSION}
 ARG R_VERSION
 ENV R_VERSION=${R_VERSION}
 
+# Old R versions (at least 3.3 and 3.4) are written in a way that is not compilable. This is likely due to the change in 
+# gcc-10 (GCC now defaults to -fno-common. As a result, global variable accesses are more efficient on various targets.
+# In C, global variables with multiple tentative definitions now result in linker errors. With -fcommon such definitions
+# are silently merged during linking.). I tried using -fcommon but without much luck so due to the time constrain I
+# decided to use gcc-9 instead.
+RUN apt-get update && apt-get install -y gcc-9 gfortran-9 g++-9 tcl-dev tk-dev && apt-get clean && apt-get autoremove -y && \
+    rm -rf /var/cache/apt/*  &&\
+    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 99 --slave /usr/bin/g++ g++ /usr/bin/g++-9 \
+    --slave /usr/bin/gcov gcov /usr/bin/gcov-9 && update-alternatives --install /usr/bin/gfortran f95 /usr/bin/gfortran-9 99
+
 COPY scripts/install_R_version /tmp/
 RUN \
     chmod +x /tmp/install_R_version && \

docker/scripts/build-h2o-3

@@ -8,7 +8,7 @@ if [[ ! ${H2O_BRANCH} ]]; then
 fi
 if [[ ! -n ${PYTHON_VERSION} ]]; then
   echo "Using default Python version"
-  PYTHON_VERSION='3.6'
+  PYTHON_VERSION='3.7'
 fi
 
 # cd home