From f981873882afad52862f24b184cb02cc4b6cc7f5 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Fri, 24 Jan 2025 16:41:20 -0500
Subject: [PATCH 01/13] Add ability to authenticate with a JWT loaded from a
file.
---
CHANGELOG.md | 4 +++-
README.md | 61 ++++++++++++++++++++++++++++++++++++++--------------
2 files changed, 48 insertions(+), 17 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 33f49d4d..22de2d02 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@
([\#50](https://github.com/ubccr/xdmod-data/pull/50),
[\#53](https://github.com/ubccr/xdmod-data/pull/53),
[\#56](https://github.com/ubccr/xdmod-data/pull/56)).
+- Add ability to authenticate with a JWT loaded from a file ([\#54](https://github.com/ubccr/xdmod-data/pull/54)).
## v1.0.2 (2024-10-31)
@@ -22,7 +23,8 @@ It is compatible with Open XDMoD versions 11.0.x and 10.5.x.
## v1.0.1 (2024-09-27)
-This release has bug fixes, performance improvements, and updates for compatibility, tests, and documentation.
+This release has bug fixes, performance improvements, and updates for
+compatibility, tests, and documentation.
It is compatible with Open XDMoD versions 11.0.x and 10.5.x.
diff --git a/README.md b/README.md
index ee5b4465..8c267c69 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,10 @@
# xdmod-data
-As part of the Data Analytics Framework for [XDMoD](https://open.xdmod.org), this Python package provides API access to the data warehouse of instances of Open XDMoD.
+As part of the Data Analytics Framework for [XDMoD](https://open.xdmod.org),
+this Python package provides API access to the data warehouse of instances of
+Open XDMoD.
-This documentation is for **v1.x.y (development branch)**. For documentation of other versions:
+This documentation is for **v1.x.y (development branch)**. For documentation of
+other versions:
- [v1.0.2](https://github.com/ubccr/xdmod-data/tree/v1.0.2?tab=readme-ov-file#xdmod-data)
- [v1.0.1](https://github.com/ubccr/xdmod-data/tree/v1.0.1?tab=readme-ov-file#xdmod-data)
@@ -12,22 +15,35 @@ The package can be installed from PyPI via `pip install xdmod-data`.
Existing installations can be upgraded via `pip install --upgrade xdmod-data`.
-The package has dependencies on [NumPy](https://pypi.org/project/numpy/), [Pandas](https://pypi.org/project/pandas/), [Plotly](https://pypi.org/project/plotly/), and [Requests](https://pypi.org/project/requests/).
+The package has dependencies on [NumPy](https://pypi.org/project/numpy/),
+[python-dotenv](https://pypi.org/project/python-dotenv/),
+[Pandas](https://pypi.org/project/pandas/),
+[Plotly](https://pypi.org/project/plotly/), and
+[Requests](https://pypi.org/project/requests/).
-Example usage is documented through Jupyter notebooks in the [xdmod-notebooks](https://github.com/ubccr/xdmod-notebooks) repository.
+Example usage is documented through Jupyter notebooks in the
+[xdmod-notebooks](https://github.com/ubccr/xdmod-notebooks) repository.
## Compatibility with Open XDMoD
-Specific versions of this package are compatible with specific versions of Open XDMoD as indicated in the table below.
+Specific versions of this package are compatible with specific versions of Open
+XDMoD as indicated in the table below.
| `xdmod-data` version | Open XDMoD versions |
| -------------------- | ------------------- |
| 1.0.2, 1.0.1 | 11.0.x, 10.5.x |
| 1.0.0 | 10.5.x |
-## API Token Access
-Use of the Data Analytics Framework requires an API token. To obtain an API token, follow the steps below to obtain an API token from the XDMoD portal.
+## API Token or JupyterHub Access
+Use of the Data Analytics Framework requires an API token, unless you are
+running in an XDMoD-hosted JupyterHub such as the one provided by ACCESS XDMoD
+(https://xdmod.access-ci.org/jupyter-hub), in which case authentication is
+handled automatically.
-1. First, if you are not already signed in to the portal, sign in in the top-left corner:
+If you need to obtain an API token, you can follow the steps below to obtain
+one from the XDMoD portal.
+
+1. First, if you are not already signed in to the portal, sign in in the
+ top-left corner:
@@ -49,17 +65,21 @@ Use of the Data Analytics Framework requires an API token. To obtain an API toke
-1. Copy the token to your clipboard. Make sure to paste it somewhere secure for saving, as you will not be able to see the token again once you close the window:
+1. Copy the token to your clipboard. Make sure to paste it somewhere secure for
+ saving, as you will not be able to see the token again once you close the
+ window:
**Note:** If you lose your token, simply delete it and generate a new one.
## Feedback / Feature Requests
-We welcome your feedback and feature requests for the Data Analytics Framework for XDMoD via email: ccr-xdmod-help@buffalo.edu.
+We welcome your feedback and feature requests for the Data Analytics Framework
+for XDMoD via email: ccr-xdmod-help@buffalo.edu.
## Support
-For support, please see [this page](https://open.xdmod.org/support.html). If you email for support, please include the following:
+For support, please see [this page](https://open.xdmod.org/support.html). If
+you email for support, please include the following:
* `xdmod-data` version number, obtained by running this Python code:
```
from xdmod_data import __version__
@@ -70,15 +90,24 @@ For support, please see [this page](https://open.xdmod.org/support.html). If you
* Detailed steps to reproduce the problem.
## License
-`xdmod-data` is released under the GNU Lesser General Public License ("LGPL") Version 3.0. See the [LICENSE](LICENSE) file for details.
+`xdmod-data` is released under the GNU Lesser General Public License ("LGPL")
+Version 3.0. See the [LICENSE](LICENSE) file for details.
## References
-When referencing the Data Analytics Framework for XDMoD, please cite the following publication:
+When referencing the Data Analytics Framework for XDMoD, please cite the
+following publication:
-> Weeden, A., White, J.P., DeLeon, R.L., Rathsam, R., Simakov, N.A., Saeli, C., and Furlani, T.R. The Data Analytics Framework for XDMoD. _SN COMPUT. SCI._ 5, 462 (2024). https://doi.org/10.1007/s42979-024-02789-2
+> Weeden, A., White, J.P., DeLeon, R.L., Rathsam, R., Simakov, N.A., Saeli, C.,
+ and Furlani, T.R. The Data Analytics Framework for XDMoD. _SN COMPUT. SCI._
+ 5, 462 (2024). https://doi.org/10.1007/s42979-024-02789-2
When referencing XDMoD, please cite the following publication:
-> Jeffrey T. Palmer, Steven M. Gallo, Thomas R. Furlani, Matthew D. Jones, Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia D. Cornelius, James C. Browne, William L. Barth, Richard T. Evans, "Open XDMoD: A Tool for the Comprehensive Management of High-Performance Computing Resources", *Computing in Science & Engineering*, Vol 17, Issue 4, 2015, pp. 52-62. DOI:[10.1109/MCSE.2015.68](https://doi.org/10.1109/MCSE.2015.68)
-
+> Jeffrey T. Palmer, Steven M. Gallo, Thomas R. Furlani, Matthew D. Jones,
+ Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette
+ Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia D. Cornelius,
+ James C. Browne, William L. Barth, Richard T. Evans, "Open XDMoD: A Tool for
+ the Comprehensive Management of High-Performance Computing Resources",
+ *Computing in Science & Engineering*, Vol 17, Issue 4, 2015, pp. 52-62.
+ DOI:[10.1109/MCSE.2015.68](https://doi.org/10.1109/MCSE.2015.68)
From 1c143e0ae0d4ce25a3b6a2be6092aa0e7527fd7b Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 14:54:53 -0500
Subject: [PATCH 02/13] Update code.
---
xdmod_data/_http_requester.py | 40 ++++++++++++++++++++++++-----------
1 file changed, 28 insertions(+), 12 deletions(-)
diff --git a/xdmod_data/_http_requester.py b/xdmod_data/_http_requester.py
index 1fd8f99f..61d9fa0b 100644
--- a/xdmod_data/_http_requester.py
+++ b/xdmod_data/_http_requester.py
@@ -1,5 +1,7 @@
+from dotenv import load_dotenv
import json
import os
+from pathlib import Path
import re
import requests
from urllib.parse import urlencode
@@ -13,14 +15,10 @@ def __init__(self, xdmod_host):
_validator._assert_str('xdmod_host', xdmod_host)
xdmod_host = re.sub('/+$', '', xdmod_host)
self.__xdmod_host = xdmod_host
- try:
+ self.__api_token = None
+ if 'XDMOD_API_TOKEN' in os.environ:
self.__api_token = os.environ['XDMOD_API_TOKEN']
- except KeyError:
- raise KeyError(
- '`XDMOD_API_TOKEN` environment variable has not been set.',
- ) from None
self.__headers = {
- 'Authorization': 'Bearer ' + self.__api_token,
'User-Agent': __title__ + ' Python v' + __version__,
}
self.__requests_session = None
@@ -124,17 +122,35 @@ def __assert_connection_to_xdmod_host(self):
def __request(self, path='', post_fields=None, stream=False):
_validator._assert_runtime_context(self.__in_runtime_context)
url = self.__xdmod_host + path
+ token_error_msg = (
+ 'If running in JupyterHub connected with XDMoD, there is likely an'
+ + ' error with the JupyterHub. Otherwise, the '
+ + ' `XDMOD_API_TOKEN` environment variable should be set'
+ + ' to a valid API token obtained from the XDMoD web portal.',
+ )
+ if self.__api_token is not None:
+ token = self.__api_token
+ else:
+ try:
+ load_dotenv(Path(os.path.expanduser('~/.xdmod-jwt.env')))
+ token = os.environ['XDMOD_JWT']
+ # TODO: add test for file not existing.
+ except KeyError:
+ raise KeyError(token_error_msg) from None
+ headers = {
+ **self.__headers,
+ **{
+ 'Authorization': 'Bearer ' + token,
+ }
+ }
if post_fields:
- post_fields['Bearer'] = self.__api_token
response = self.__requests_session.post(
url,
- headers=self.__headers,
+ headers=headers,
data=post_fields,
)
else:
- url += '&' if '?' in url else '?'
- url += 'Bearer=' + self.__api_token
- response = self.__requests_session.get(url, headers=self.__headers)
+ response = self.__requests_session.get(url, headers=headers)
if response.status_code != 200:
msg = ''
try:
@@ -144,7 +160,7 @@ def __request(self, path='', post_fields=None, stream=False):
pass
if response.status_code == 401:
msg = (
- ': Make sure XDMOD_API_TOKEN is set to a valid API token.'
+ ': ' + token_error_msg
)
raise RuntimeError(
'Error ' + str(response.status_code) + msg,
From e8f570a038ed1a5bfd1c7994defe5b0b4b1fc69b Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 14:59:14 -0500
Subject: [PATCH 03/13] Fix linter.
---
xdmod_data/_http_requester.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/xdmod_data/_http_requester.py b/xdmod_data/_http_requester.py
index 61d9fa0b..daee43d1 100644
--- a/xdmod_data/_http_requester.py
+++ b/xdmod_data/_http_requester.py
@@ -141,7 +141,7 @@ def __request(self, path='', post_fields=None, stream=False):
**self.__headers,
**{
'Authorization': 'Bearer ' + token,
- }
+ },
}
if post_fields:
response = self.__requests_session.post(
From 83e283e76d43adb610cde50aa41796944b2dd553 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 15:33:58 -0500
Subject: [PATCH 04/13] Document and debug.
---
tests/unit/test_datawarehouse_unit.py | 20 +++++--------
xdmod_data/_http_requester.py | 9 +++---
xdmod_data/warehouse.py | 43 +++++++++++++++++++--------
3 files changed, 42 insertions(+), 30 deletions(-)
diff --git a/tests/unit/test_datawarehouse_unit.py b/tests/unit/test_datawarehouse_unit.py
index 9627e0f1..7bc8fdb3 100644
--- a/tests/unit/test_datawarehouse_unit.py
+++ b/tests/unit/test_datawarehouse_unit.py
@@ -25,17 +25,6 @@ def test___init___TypeError_xdmod_host():
DataWarehouse(2)
-def test___init___KeyError():
- token = os.environ['XDMOD_API_TOKEN']
- del os.environ['XDMOD_API_TOKEN']
- with pytest.raises(
- KeyError,
- match='`XDMOD_API_TOKEN` environment variable has not been set.',
- ):
- DataWarehouse(VALID_XDMOD_HOST)
- os.environ['XDMOD_API_TOKEN'] = token
-
-
def test___enter___RuntimeError_xdmod_host_malformed():
with pytest.raises(
(
@@ -73,8 +62,13 @@ def test___enter___RuntimeError_xdmod_host_unsupported_protocol():
def test___enter___RuntimeError_401():
with pytest.raises(
RuntimeError,
- match='Error 401: Make sure XDMOD_API_TOKEN is set'
- + ' to a valid API token.',
+ match=(
+ 'Error 401: If running in JupyterHub connected with XDMoD, this'
+ + ' is likely an error with the JupyterHub. Otherwise, make sure'
+ + ' the `XDMOD_API_TOKEN` environment variable is set before the'
+ + ' `DataWarehouse` is constructed; it should be set to a valid'
+ + ' API token obtained from the XDMoD web portal.'
+ )
):
with DataWarehouse(VALID_XDMOD_HOST) as dw:
dw.describe_realms()
diff --git a/xdmod_data/_http_requester.py b/xdmod_data/_http_requester.py
index daee43d1..45e75387 100644
--- a/xdmod_data/_http_requester.py
+++ b/xdmod_data/_http_requester.py
@@ -123,10 +123,11 @@ def __request(self, path='', post_fields=None, stream=False):
_validator._assert_runtime_context(self.__in_runtime_context)
url = self.__xdmod_host + path
token_error_msg = (
- 'If running in JupyterHub connected with XDMoD, there is likely an'
- + ' error with the JupyterHub. Otherwise, the '
- + ' `XDMOD_API_TOKEN` environment variable should be set'
- + ' to a valid API token obtained from the XDMoD web portal.',
+ 'If running in JupyterHub connected with XDMoD, this is likely an'
+ + ' error with the JupyterHub. Otherwise, make sure the'
+ + ' `XDMOD_API_TOKEN` environment variable is set before the'
+ + ' `DataWarehouse` is constructed; it should be set to a valid'
+ + ' API token obtained from the XDMoD web portal.'
)
if self.__api_token is not None:
token = self.__api_token
diff --git a/xdmod_data/warehouse.py b/xdmod_data/warehouse.py
index c1c56945..8a0082bd 100644
--- a/xdmod_data/warehouse.py
+++ b/xdmod_data/warehouse.py
@@ -15,6 +15,12 @@ class DataWarehouse:
>>> with DataWarehouse('https://xdmod.access-ci.org') as dw:
... dw.get_data()
+ If running in a JupyterHub connected to XDMoD, authentication should
+ happen automatically. Otherwise, make sure the `XDMOD_API_TOKEN`
+ environment variable is set before the `DataWarehouse` is
+ constructed; it should be set to a valid API token obtained from
+ the XDMoD web portal.
+
Parameters
----------
xdmod_host : str
@@ -22,8 +28,6 @@ class DataWarehouse:
Raises
------
- KeyError
- If the `XDMOD_API_TOKEN` environment variable has not been set.
RuntimeError
If a connection cannot be made to the XDMoD server specified by
`xdmod_host`.
@@ -112,8 +116,9 @@ def get_data(
Raises
------
KeyError
- If any of the parameters have invalid values. Valid realms
- come from `describe_realms()`, valid metrics come from
+ If an authentication token could not be loaded or if any of the
+ parameters have invalid values. Valid realms come from
+ `describe_realms()`, valid metrics come from
`describe_metrics()`, valid dimensions and filter keys come from
`describe_dimensions()`, valid filter values come from
`get_filter_values()`, valid durations come from
@@ -182,8 +187,9 @@ def get_raw_data(
Raises
------
KeyError
- If any of the parameters have invalid values. Valid durations
- come from `get_durations()`, valid realms come from
+ If an authentication token could not be loaded or if any of the
+ parameters have invalid values. Valid durations come from
+ `get_durations()`, valid realms come from
`describe_raw_realms()`, valid filters keys come from
`describe_dimensions()`, valid filter values come from
`get_filter_values()`, and valid fields come from
@@ -215,6 +221,8 @@ def describe_realms(self):
Raises
------
+ KeyError
+ If an authentication token could not be loaded.
RuntimeError
If this method is called outside the runtime context.
"""
@@ -243,7 +251,8 @@ def describe_metrics(self, realm):
Raises
------
KeyError
- If `realm` is not one of the values from `describe_realms()`.
+ If an authentication token could not be loaded or if `realm` is
+ not one of the values from `describe_realms()`.
RuntimeError
If this method is called outside the runtime context.
TypeError
@@ -270,7 +279,8 @@ def describe_dimensions(self, realm):
Raises
------
KeyError
- If `realm` is not one of the values from `describe_realms()`.
+ If an authentication token could not be loaded or if `realm` is
+ not one of the values from `describe_realms()`.
RuntimeError
If this method is called outside the runtime context.
TypeError
@@ -300,9 +310,9 @@ def get_filter_values(self, realm, dimension):
Raises
------
KeyError
- If `realm` is not one of the values from `describe_realms()` or
- `dimension` is not one of the IDs or labels from
- `describe_dimensions()`.
+ If an authentication token could not be loaded or if `realm` is
+ not one of the values from `describe_realms()` or `dimension` is
+ not one of the IDs or labels from `describe_dimensions()`.
RuntimeError
If this method is called outside the runtime context.
TypeError
@@ -330,6 +340,11 @@ def get_durations(self):
Returns
-------
tuple of str
+
+ Raises
+ ------
+ KeyError
+ If an authentication token could not be loaded or if `realm` is
"""
return _validator._get_durations()
@@ -355,6 +370,8 @@ def describe_raw_realms(self):
Raises
------
+ KeyError
+ If an authentication token could not be loaded.
RuntimeError
If this method is called outside the runtime context.
"""
@@ -383,8 +400,8 @@ def describe_raw_fields(self, realm):
Raises
------
KeyError
- If `realm` is not one of the values from
- `describe_raw_realms()`.
+ If an authentication token could not be loaded or if `realm` is
+ not one of the values from `describe_raw_realms()`.
RuntimeError
If this method is called outside the runtime context.
TypeError
From bc67ab00f0a4cd87dd362da774694c5c4b6457e9 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 15:37:03 -0500
Subject: [PATCH 05/13] Update docs and setup.cfg, lint.
---
README.md | 4 ++--
setup.cfg | 1 +
tests/unit/test_datawarehouse_unit.py | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index 8c267c69..2b39b10e 100644
--- a/README.md
+++ b/README.md
@@ -16,9 +16,9 @@ The package can be installed from PyPI via `pip install xdmod-data`.
Existing installations can be upgraded via `pip install --upgrade xdmod-data`.
The package has dependencies on [NumPy](https://pypi.org/project/numpy/),
-[python-dotenv](https://pypi.org/project/python-dotenv/),
[Pandas](https://pypi.org/project/pandas/),
-[Plotly](https://pypi.org/project/plotly/), and
+[Plotly](https://pypi.org/project/plotly/),
+[Python-dotenv](https://pypi.org/project/python-dotenv/), and
[Requests](https://pypi.org/project/requests/).
Example usage is documented through Jupyter notebooks in the
diff --git a/setup.cfg b/setup.cfg
index f87aef0f..3f06c469 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -17,4 +17,5 @@ install_requires =
numpy >= 1.23.0
pandas >= 1.5.0
plotly >= 5.8.0
+ python-requests >= 1.0.0
requests >= 2.19.0
diff --git a/tests/unit/test_datawarehouse_unit.py b/tests/unit/test_datawarehouse_unit.py
index 7bc8fdb3..264e2768 100644
--- a/tests/unit/test_datawarehouse_unit.py
+++ b/tests/unit/test_datawarehouse_unit.py
@@ -68,7 +68,7 @@ def test___enter___RuntimeError_401():
+ ' the `XDMOD_API_TOKEN` environment variable is set before the'
+ ' `DataWarehouse` is constructed; it should be set to a valid'
+ ' API token obtained from the XDMoD web portal.'
- )
+ ),
):
with DataWarehouse(VALID_XDMOD_HOST) as dw:
dw.describe_realms()
From 495d5a0a927a5851aaa6a468b0d0e601804c1580 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 15:39:26 -0500
Subject: [PATCH 06/13] Debug.
---
setup.cfg | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/setup.cfg b/setup.cfg
index 3f06c469..d3481479 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -17,5 +17,5 @@ install_requires =
numpy >= 1.23.0
pandas >= 1.5.0
plotly >= 5.8.0
- python-requests >= 1.0.0
+ python-dotenv >= 1.0.0
requests >= 2.19.0
From 20d0bf384b17b3f7875227bd294499988237479a Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 16:25:34 -0500
Subject: [PATCH 07/13] Add partially working test, update version number.
---
.../test_datawarehouse_integration.py | 55 ++++++++++++++++---
1 file changed, 46 insertions(+), 9 deletions(-)
diff --git a/tests/integration/test_datawarehouse_integration.py b/tests/integration/test_datawarehouse_integration.py
index abeaf8fa..cf61bd66 100644
--- a/tests/integration/test_datawarehouse_integration.py
+++ b/tests/integration/test_datawarehouse_integration.py
@@ -65,13 +65,13 @@
'field': (INVALID_STR, r'Field .* not found'),
}
-key_error_test_ids = []
+param_key_error_test_ids = []
duration_test_ids = []
start_end_test_ids = []
type_error_test_ids = []
default_valid_params = {}
-key_error_test_params = []
+param_key_error_test_params = []
date_malformed_test_params = []
type_error_test_params = []
value_error_test_methods = []
@@ -83,9 +83,9 @@
type_error_test_ids += [method + ':' + param]
type_error_test_params += [(method, param)]
if param in KEY_ERROR_TEST_VALUES_AND_MATCHES:
- key_error_test_ids += [method + ':' + param]
+ param_key_error_test_ids += [method + ':' + param]
(value, match) = KEY_ERROR_TEST_VALUES_AND_MATCHES[param]
- key_error_test_params += [(method, {param: value}, match)]
+ param_key_error_test_params += [(method, {param: value}, match)]
if param == 'duration':
duration_test_ids += [method]
start_end_test_ids += [
@@ -107,9 +107,13 @@
value_error_test_methods += [method]
if 'filters' in METHOD_PARAMS[method]:
for param in ('filter_key', 'filter_value'):
- key_error_test_ids += [method + ':' + param]
+ param_key_error_test_ids += [method + ':' + param]
(value, match) = KEY_ERROR_TEST_VALUES_AND_MATCHES[param]
- key_error_test_params += [(method, {'filters': value}, match)]
+ param_key_error_test_params += [(
+ method,
+ {'filters': value},
+ match,
+ )]
load_dotenv(Path(os.path.expanduser(TOKEN_PATH)), override=True)
@@ -130,6 +134,19 @@ def dw_methods_outside_runtime_context():
return __get_dw_methods(dw)
+@pytest.fixture(scope='module')
+def dw_methods_no_xdmod_api_token():
+ token = os.environ['XDMOD_API_TOKEN']
+ del os.environ['XDMOD_API_TOKEN']
+ with open(Path(os.path.expanduser('~/.xdmod-jwt.env')), 'w') as jwt_file:
+ jwt_file.write('XDMOD_JWT=' + token + '\n')
+ dw = DataWarehouse(VALID_XDMOD_HOST)
+ os.environ['XDMOD_API_TOKEN'] = token
+ with dw:
+ os.remove(Path(os.path.expanduser('~/.xdmod-jwt.env')))
+ yield __get_dw_methods(dw)
+
+
def __get_dw_methods(dw):
return {
'get_data': dw.get_data,
@@ -153,12 +170,32 @@ def __test_exception(dw_methods, method, additional_params, error, match):
__run_method(dw_methods, method, additional_params)
+@pytest.mark.parametrize(
+ 'method',
+ list(METHOD_PARAMS.keys()),
+)
+def test_token_KeyError(dw_methods_no_xdmod_api_token, method):
+ __test_exception(
+ dw_methods_no_xdmod_api_token,
+ method,
+ {},
+ KeyError,
+ (
+ 'If running in JupyterHub connected with XDMoD, this is likely an'
+ + ' error with the JupyterHub. Otherwise, make sure the'
+ + ' `XDMOD_API_TOKEN` environment variable is set before the'
+ + ' `DataWarehouse` is constructed; it should be set to a valid'
+ + ' API token obtained from the XDMoD web portal.'
+ ),
+ )
+
+
@pytest.mark.parametrize(
'method, params, match',
- key_error_test_params,
- ids=key_error_test_ids,
+ param_key_error_test_params,
+ ids=param_key_error_test_ids,
)
-def test_KeyError(dw_methods, method, params, match):
+def test_param_KeyError(dw_methods, method, params, match):
__test_exception(dw_methods, method, params, KeyError, match)
From 8440cfb3c863c296637e395b139f2e4b17906f99 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 17:41:02 -0500
Subject: [PATCH 08/13] Debug.
---
xdmod_data/_http_requester.py | 7 +++----
xdmod_data/warehouse.py | 5 -----
2 files changed, 3 insertions(+), 9 deletions(-)
diff --git a/xdmod_data/_http_requester.py b/xdmod_data/_http_requester.py
index 45e75387..914fd54a 100644
--- a/xdmod_data/_http_requester.py
+++ b/xdmod_data/_http_requester.py
@@ -1,4 +1,4 @@
-from dotenv import load_dotenv
+from dotenv import dotenv_values
import json
import os
from pathlib import Path
@@ -133,9 +133,8 @@ def __request(self, path='', post_fields=None, stream=False):
token = self.__api_token
else:
try:
- load_dotenv(Path(os.path.expanduser('~/.xdmod-jwt.env')))
- token = os.environ['XDMOD_JWT']
- # TODO: add test for file not existing.
+ values = dotenv_values(Path(os.path.expanduser('~/.xdmod-jwt.env')))
+ token = values['XDMOD_JWT']
except KeyError:
raise KeyError(token_error_msg) from None
headers = {
diff --git a/xdmod_data/warehouse.py b/xdmod_data/warehouse.py
index 8a0082bd..363aaf25 100644
--- a/xdmod_data/warehouse.py
+++ b/xdmod_data/warehouse.py
@@ -340,11 +340,6 @@ def get_durations(self):
Returns
-------
tuple of str
-
- Raises
- ------
- KeyError
- If an authentication token could not be loaded or if `realm` is
"""
return _validator._get_durations()
From 7f3c607f7bf4d327e6764962356b3428876cbac3 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 17:43:32 -0500
Subject: [PATCH 09/13] Lint.
---
xdmod_data/_http_requester.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/xdmod_data/_http_requester.py b/xdmod_data/_http_requester.py
index 914fd54a..a65e3a44 100644
--- a/xdmod_data/_http_requester.py
+++ b/xdmod_data/_http_requester.py
@@ -133,7 +133,9 @@ def __request(self, path='', post_fields=None, stream=False):
token = self.__api_token
else:
try:
- values = dotenv_values(Path(os.path.expanduser('~/.xdmod-jwt.env')))
+ values = dotenv_values(
+ Path(os.path.expanduser('~/.xdmod-jwt.env')),
+ )
token = values['XDMOD_JWT']
except KeyError:
raise KeyError(token_error_msg) from None
From 3e42fa646a404cdc634d84ea4bd1eedfd3aa9021 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Mon, 27 Jan 2025 17:52:29 -0500
Subject: [PATCH 10/13] Use invalid string for default JWT in integration
testing.
---
tests/integration/test_datawarehouse_integration.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/integration/test_datawarehouse_integration.py b/tests/integration/test_datawarehouse_integration.py
index cf61bd66..758c57f5 100644
--- a/tests/integration/test_datawarehouse_integration.py
+++ b/tests/integration/test_datawarehouse_integration.py
@@ -139,7 +139,7 @@ def dw_methods_no_xdmod_api_token():
token = os.environ['XDMOD_API_TOKEN']
del os.environ['XDMOD_API_TOKEN']
with open(Path(os.path.expanduser('~/.xdmod-jwt.env')), 'w') as jwt_file:
- jwt_file.write('XDMOD_JWT=' + token + '\n')
+ jwt_file.write('XDMOD_JWT=' + INVALID_STR + '\n')
dw = DataWarehouse(VALID_XDMOD_HOST)
os.environ['XDMOD_API_TOKEN'] = token
with dw:
From 01c983615083267f4ccdb09fed64becd4d3a5742 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Tue, 28 Jan 2025 11:02:22 -0500
Subject: [PATCH 11/13] Remove redundant dependency installation.
---
.circleci/config.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.circleci/config.yml b/.circleci/config.yml
index dc897f31..31c35cec 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -28,7 +28,7 @@ jobs:
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade flake8 flake8-commas flake8-quotes'
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py'
docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install -e .'
- docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade python-dotenv pytest coverage'
+ docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade pytest coverage'
# The minimum version of each dependency should be tested in the
# container with the minimum Python version.
if [ "$python_container" = "python-min" ]; then
From d681f0f78af56bc6de6958d3b2292b2b9da3f19d Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Wed, 29 Jan 2025 10:57:42 -0500
Subject: [PATCH 12/13] Fix test.
---
tests/unit/test_datawarehouse_unit.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/tests/unit/test_datawarehouse_unit.py b/tests/unit/test_datawarehouse_unit.py
index 264e2768..2b7a548f 100644
--- a/tests/unit/test_datawarehouse_unit.py
+++ b/tests/unit/test_datawarehouse_unit.py
@@ -33,9 +33,8 @@ def test___enter___RuntimeError_xdmod_host_malformed():
),
match=(
r'(Invalid URL \'.*\': No host supplied|'
- + r'Invalid URL \'https:\?Bearer=' + INVALID_STR + "': "
- + r'No schema supplied. Perhaps you meant http://https:\?Bearer='
- + INVALID_STR + r'\?)'
+ + r'Invalid URL \'https:\': No schema supplied.'
+ + r' Perhaps you meant http://https:\?'
),
):
with DataWarehouse('https://'): # pragma: no cover
From e564c4da138b7d9db1e92853730e35508b8e1c25 Mon Sep 17 00:00:00 2001
From: Aaron Weeden <aaronwee@buffalo.edu>
Date: Wed, 29 Jan 2025 11:44:00 -0500
Subject: [PATCH 13/13] Fix syntax error.
---
tests/unit/test_datawarehouse_unit.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/unit/test_datawarehouse_unit.py b/tests/unit/test_datawarehouse_unit.py
index 2b7a548f..0111c062 100644
--- a/tests/unit/test_datawarehouse_unit.py
+++ b/tests/unit/test_datawarehouse_unit.py
@@ -34,7 +34,7 @@ def test___enter___RuntimeError_xdmod_host_malformed():
match=(
r'(Invalid URL \'.*\': No host supplied|'
+ r'Invalid URL \'https:\': No schema supplied.'
- + r' Perhaps you meant http://https:\?'
+ + r' Perhaps you meant http://https:\?)'
),
):
with DataWarehouse('https://'): # pragma: no cover