Skip to content

Commit fe1c967

Browse files
aaronweedenRodneyCodess
aaronweeden
and
RodneyCodess
committed
Set up Docker Compose for CI testing. (#53)
Co-authored-by: Rodney Garnett <[email protected]>
1 parent a90fc5c commit fe1c967

35 files changed

+10617
-580
lines changed

‎.circleci/config.yml

+105-27
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,113 @@
11
version: 2.1
22
jobs:
33
build:
4-
parameters:
5-
python-version:
6-
type: string
4+
environment:
5+
MIN_PYTHON_VERSION: "3.8"
6+
MAX_PYTHON_VERSION: "3.13"
7+
XDMOD_10_5_IMAGE: tools-ext-01.ccr.xdmod.org/xdmod-10.5.0-x86_64:rockylinux8.5-0.3
8+
XDMOD_11_0_IMAGE: tools-ext-01.ccr.xdmod.org/xdmod:x86_64-rockylinux8.9.20231119-v11.0.0-1.0-03
79
docker:
8-
- image: cimg/python:<< parameters.python-version >>
10+
- image: cimg/base:current
911
steps:
1012
- checkout
13+
- setup_remote_docker
1114
- run:
12-
name: Install dependencies
15+
name: Spin up containers and test different Python versions against different XDMoD web server versions
1316
command: |
14-
python -m pip install --upgrade pip
15-
python -m pip install flake8 flake8-commas flake8-quotes pytest
16-
python -m pip install -e .
17-
- run:
18-
name: Lint with Flake8
19-
command: python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py
20-
- run:
21-
name: Test with pytest
22-
command: python3 -m pytest tests/unit
23-
workflows:
24-
full-build:
25-
jobs:
26-
- build:
27-
matrix:
28-
parameters:
29-
python-version:
30-
- "3.13"
31-
- "3.12"
32-
- "3.11"
33-
- "3.10"
34-
- "3.9"
35-
- "3.8"
17+
set -x
18+
docker compose -f .circleci/docker-compose.yml up -d
19+
declare -a python_containers=$(yq '.services | keys | .[] | select(. == "python-*")' .circleci/docker-compose.yml)
20+
declare -a xdmod_containers=$(yq '.services | keys | .[] | select(. == "xdmod-*")' .circleci/docker-compose.yml)
21+
# Copy the xdmod-data source code to the Python containers, lint
22+
# with Flake 8, and install the package and its testing
23+
# dependencies.
24+
for python_container in $python_containers; do
25+
docker cp . $python_container:/home/circleci/project
26+
docker exec $python_container bash -c 'sudo chown -R circleci:circleci /home/circleci/project'
27+
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade pip'
28+
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade flake8 flake8-commas flake8-quotes'
29+
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py'
30+
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install -e .'
31+
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade python-dotenv pytest coverage'
32+
done
33+
# Set up XDMoD web server containers.
34+
for xdmod_container in $xdmod_containers; do
35+
# Generate OpenSSL key and certificate.
36+
docker exec $xdmod_container bash -c "openssl genrsa -rand /proc/cpuinfo:/proc/filesystems:/proc/interrupts:/proc/ioports:/proc/uptime 2048 > /etc/pki/tls/private/$xdmod_container.key"
37+
docker exec $xdmod_container bash -c "openssl req -new -key /etc/pki/tls/private/$xdmod_container.key -x509 -sha256 -days 365 -set_serial $RANDOM -extensions v3_req -out /etc/pki/tls/certs/$xdmod_container.crt -subj '/C=XX/L=Default City/O=Default Company Ltd/CN=$xdmod_container' -addext 'subjectAltName=DNS:$xdmod_container'"
38+
# Update the server hostnames and certificates so the Python
39+
# containers can make requests to them.
40+
docker exec $xdmod_container bash -c "sed -i \"s/localhost/$xdmod_container/g\" /etc/httpd/conf.d/xdmod.conf"
41+
if [[ "$xdmod_container" =~ 'xdmod-*-dev' ]]; then
42+
if [ "$xdmod_container" = 'xdmod-main-dev' ]; then
43+
branch='main'
44+
else
45+
branch="xdmod$(echo $xdmod_container | sed 's/xdmod-\(.*\)-dev/\1/' | sed 's/-/./')"
46+
fi
47+
# Install and run the latest development version of the XDMoD
48+
# web server.
49+
docker exec $xdmod_container bash -c 'git clone --depth=1 --branch=$branch https://github.com/ubccr/xdmod.git /root/xdmod'
50+
docker exec -w /root/xdmod $xdmod_container bash -c 'composer install'
51+
docker exec -w /root/xdmod $xdmod_container bash -c '/root/bin/buildrpm xdmod'
52+
docker exec -w /root/xdmod $xdmod_container bash -c 'XDMOD_TEST_MODE=upgrade ./tests/ci/bootstrap.sh'
53+
docker exec -w /root/xdmod $xdmod_container bash -c './tests/ci/validate.sh'
54+
elif [[ "$xdmod_container" =~ xdmod-* ]]; then
55+
# Run the XDMoD web server.
56+
docker exec $xdmod_container bash -c '/root/bin/services start'
57+
fi
58+
# Copy the 10,000 users file into the container and shred it.
59+
# We use this file so we can test filters with more than 10,000
60+
# values and date ranges that span multiple quarters.
61+
docker cp tests/ci/artifacts/10000users.log $xdmod_container:.
62+
docker exec $xdmod_container xdmod-shredder -r frearson -f slurm -i 10000users.log
63+
# Ingest and aggregate.
64+
date=$(date --utc +%Y-%m-%d)
65+
docker exec $xdmod_container xdmod-ingestor --ingest
66+
docker exec $xdmod_container xdmod-ingestor --aggregate=job --last-modified-start-date $date
67+
# Copy certificate (for doing requests) from the XDMoD container.
68+
docker cp $xdmod_container:/etc/pki/tls/certs/$xdmod_container.crt .
69+
# Copy certificate to one of the Python containers and get an
70+
# XDMoD API token for the XDMoD container.
71+
docker cp $xdmod_container.crt $python_container:/home/circleci/project
72+
rest_token=$(docker exec \
73+
-e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
74+
$python_container \
75+
bash -c "curl \
76+
-sS \
77+
-X POST \
78+
-c xdmod.cookie \
79+
-d 'username=normaluser&password=normaluser' \
80+
https://$xdmod_container/rest/auth/login \
81+
| jq -r '.results.token'"
82+
)
83+
docker exec $python_container bash -c 'echo -n "XDMOD_API_TOKEN="' > ${xdmod_container}-token
84+
docker exec \
85+
-e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
86+
$python_container \
87+
bash -c "curl \
88+
-sS \
89+
-X POST \
90+
-b xdmod.cookie \
91+
https://$xdmod_container/rest/users/current/api/token?token=$rest_token \
92+
| jq -r '.data.token'" \
93+
>> ${xdmod_container}-token
94+
done
95+
# Run the tests against each XDMoD web server.
96+
for python_container in $python_containers; do
97+
for xdmod_container in $xdmod_containers; do
98+
# Copy certificate (for doing requests) to the Python
99+
# container.
100+
docker cp $xdmod_container.crt $python_container:/home/circleci/project
101+
# Copy XDMoD API token to the Python container.
102+
docker cp ${xdmod_container}-token $python_container:/home/circleci/.xdmod-data-token
103+
# Run tests in the Python container.
104+
docker exec \
105+
-e CURL_CA_BUNDLE="/home/circleci/project/$xdmod_container.crt" \
106+
-e XDMOD_HOST="https://$xdmod_container" \
107+
-e XDMOD_VERSION="$xdmod_container" \
108+
$python_container \
109+
bash -c 'python3 -m coverage run --branch --append -m pytest -vvs -o log_cli=true tests/'
110+
done
111+
# Make sure 100% test coverage.
112+
docker exec $python_container bash -c 'python3 -m coverage report -m --fail-under=100'
113+
done

‎.circleci/docker-compose.yml

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
services:
2+
# Test the minimum and maximum supported Python versions:
3+
python-min:
4+
container_name: python-min
5+
image: cimg/python:${MIN_PYTHON_VERSION}
6+
tty: true
7+
python-max:
8+
container_name: python-max
9+
image: cimg/python:${MAX_PYTHON_VERSION}
10+
tty: true
11+
# Test the supported and development versions of the XDMoD web server:
12+
xdmod-main-dev:
13+
container_name: xdmod-main-dev
14+
image: ${XDMOD_11_0_IMAGE}
15+
tty: true
16+
xdmod-11-0-dev:
17+
container_name: xdmod-11-0-dev
18+
image: ${XDMOD_11_0_IMAGE}
19+
tty: true
20+
xdmod-11-0:
21+
container_name: xdmod-11-0
22+
image: ${XDMOD_11_0_IMAGE}
23+
tty: true
24+
xdmod-10-5:
25+
container_name: xdmod-10-5
26+
image: ${XDMOD_10_5_IMAGE}
27+
tty: true

‎.github/PULL_REQUEST_TEMPLATE.md

-1
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,5 @@
3030
- [ ] `xdmod_data/__version__.py` has been updated to the next development version
3131
- [ ] The milestone is set correctly on the pull request
3232
- [ ] The appropriate labels have been added to the pull request
33-
- [ ] Running the automated tests (see `docs/developing.md`) produces no errors
3433
- [ ] Updates have been made to the `xdmod-notebooks` repository as necessary, and the notebooks all run successfully
3534
- [ ] The changes in this PR have been ported/backported to other branches as needed

‎.gitignore

+4-129
Original file line numberDiff line numberDiff line change
@@ -1,132 +1,7 @@
1-
# Byte-compiled / optimized / DLL files
2-
__pycache__/
3-
*.py[cod]
4-
*$py.class
5-
6-
# C extensions
7-
*.so
8-
9-
# Distribution / packaging
10-
.Python
111
build/
12-
develop-eggs/
13-
dist/
14-
downloads/
15-
eggs/
16-
.eggs/
17-
lib/
18-
lib64/
19-
parts/
20-
sdist/
21-
var/
22-
wheels/
23-
pip-wheel-metadata/
24-
share/python-wheels/
25-
*.egg-info/
26-
.installed.cfg
27-
*.egg
28-
MANIFEST
29-
30-
# PyInstaller
31-
# Usually these files are written by a python script from a template
32-
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33-
*.manifest
34-
*.spec
35-
36-
# Installer logs
37-
pip-log.txt
38-
pip-delete-this-directory.txt
39-
40-
# Unit test / coverage reports
41-
htmlcov/
42-
.tox/
43-
.nox/
442
.coverage
45-
.coverage.*
46-
.cache
47-
nosetests.xml
48-
coverage.xml
49-
*.cover
50-
*.py,cover
51-
.hypothesis/
52-
.pytest_cache/
53-
54-
# Translations
55-
*.mo
56-
*.pot
57-
58-
# Django stuff:
59-
*.log
60-
local_settings.py
61-
db.sqlite3
62-
db.sqlite3-journal
63-
64-
# Flask stuff:
65-
instance/
66-
.webassets-cache
67-
68-
# Scrapy stuff:
69-
.scrapy
70-
71-
# Sphinx documentation
3+
dist/
724
docs/_build/
73-
74-
# PyBuilder
75-
target/
76-
77-
# Jupyter Notebook
78-
.ipynb_checkpoints
79-
80-
# IPython
81-
profile_default/
82-
ipython_config.py
83-
84-
# pyenv
85-
.python-version
86-
87-
# pipenv
88-
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89-
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90-
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91-
# install all needed dependencies.
92-
#Pipfile.lock
93-
94-
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95-
__pypackages__/
96-
97-
# Celery stuff
98-
celerybeat-schedule
99-
celerybeat.pid
100-
101-
# SageMath parsed files
102-
*.sage.py
103-
104-
# Environments
105-
.env
106-
.venv
107-
env/
108-
venv/
109-
ENV/
110-
env.bak/
111-
venv.bak/
112-
113-
# Spyder project settings
114-
.spyderproject
115-
.spyproject
116-
117-
# Rope project settings
118-
.ropeproject
119-
120-
# mkdocs documentation
121-
/site
122-
123-
# mypy
124-
.mypy_cache/
125-
.dmypy.json
126-
dmypy.json
127-
128-
# Pyre type checker
129-
.pyre/
130-
131-
# Vi swap files
132-
.*.swp
5+
*.egg-info/
6+
__pycache__/
7+
.*.sw?

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
- Implement 100% test coverage ([\#45](https://github.com/ubccr/xdmod-data/pull/45)).
77
- Update Flake8 rules ([\#46](https://github.com/ubccr/xdmod-data/pull/46)).
88
- Change to using CircleCI instead of GitHub Actions for continuous integration ([\#50](https://github.com/ubccr/xdmod-data/pull/50)).
9+
- Set up Docker Compose for CI testing ([\#53](https://github.com/ubccr/xdmod-data/pull/53)).
910

1011
## v1.0.2 (2024-10-31)
1112

‎docs/developing.md

-59
Original file line numberDiff line numberDiff line change
@@ -1,64 +1,5 @@
11
# Instructions for developers
22

3-
## Testing the code
4-
1. Start up a virtual environment, e.g.:
5-
```
6-
python3 -m venv ~/xdmod-data-test-env
7-
source ~/xdmod-data-test-env/bin/activate
8-
```
9-
Your command prompt should now start with `(xdmod-data-test-env)`.
10-
1. Install your local development copy of `xdmod-data` in editable mode:
11-
```
12-
python3 -m pip install --force-reinstall -e /path/to/your/xdmod-data
13-
```
14-
1. Install `python-dotenv`, `pytest`, and `coverage`:
15-
```
16-
python3 -m pip install --upgrade python-dotenv pytest coverage
17-
```
18-
1. Create an empty file in your home directory at `~/.xdmod-data-token` with permissions set to 600.
19-
1. With an [ACCESS XDMoD](https://xdmod.access-ci.org) account with "User" as the Top Role, create an API token if you do not already have one (sign in and click My Profile -> API Token).
20-
1. Add the following line to the file `~/.xdmod-data-token`, replacing `<token>` with your token.
21-
```
22-
XDMOD_API_TOKEN=<token>
23-
```
24-
1. Change directories to your local development copy of `xdmod-data`.
25-
1. Run the following command and make sure all the tests pass:
26-
```
27-
coverage run -m pytest -vvs -o log_cli=true tests/
28-
```
29-
1. Run the following command and make sure the code is 100% covered by tests:
30-
```
31-
coverage report -m
32-
```
33-
1. Downgrade to the minimum version of the dependencies. Replace the version numbers below with the values from `setup.cfg`.
34-
```
35-
python3 -m pip install --force-reinstall numpy==1.23.0 pandas==1.5.0 plotly==5.8.0 requests==2.19.0
36-
```
37-
1. Run the following command again and make sure all the tests pass (Deprecation warnings in `urllib3` are OK).
38-
```
39-
coverage run -m pytest -vvs -o log_cli=true tests/
40-
```
41-
1. Run the following command and make sure the code is 100% covered by tests:
42-
```
43-
coverage report -m
44-
```
45-
46-
## Linting the code
47-
1. Start up a virtual environment, e.g.:
48-
```
49-
python3 -m venv ~/xdmod-data-test-env
50-
source ~/xdmod-data-test-env/bin/activate
51-
```
52-
Your command prompt should now start with `(xdmod-data-test-env)`.
53-
1. Install Flake8 and additional packages
54-
```
55-
python3 -m pip install flake8 flake8-quotes flake8-commas
56-
```
57-
1. Run Flake8
58-
```
59-
flake8 --max-complexity=10 --show-source --exclude __init__.py /path/to/your/xdmod-data
60-
```
61-
623
## Releasing a new version
634
1. Make a new branch of `xdmod-data` and:
645
1. Make sure the version number is updated in `xdmod_data/__version__.py`.

0 commit comments

Comments
 (0)