diff --git a/.circleci/config.yml b/.circleci/config.yml index dc897f31..31c35cec 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -28,7 +28,7 @@ jobs: docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade flake8 flake8-commas flake8-quotes' docker exec -w /home/circleci/project $python_container bash -c 'python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py' docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install -e .' - docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade python-dotenv pytest coverage' + docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade pytest coverage' # The minimum version of each dependency should be tested in the # container with the minimum Python version. if [ "$python_container" = "python-min" ]; then diff --git a/CHANGELOG.md b/CHANGELOG.md index 33f49d4d..22de2d02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ ([\#50](https://github.com/ubccr/xdmod-data/pull/50), [\#53](https://github.com/ubccr/xdmod-data/pull/53), [\#56](https://github.com/ubccr/xdmod-data/pull/56)). +- Add ability to authenticate with a JWT loaded from a file ([\#54](https://github.com/ubccr/xdmod-data/pull/54)). ## v1.0.2 (2024-10-31) @@ -22,7 +23,8 @@ It is compatible with Open XDMoD versions 11.0.x and 10.5.x. ## v1.0.1 (2024-09-27) -This release has bug fixes, performance improvements, and updates for compatibility, tests, and documentation. +This release has bug fixes, performance improvements, and updates for +compatibility, tests, and documentation. It is compatible with Open XDMoD versions 11.0.x and 10.5.x. diff --git a/README.md b/README.md index ee5b4465..2b39b10e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,10 @@ # xdmod-data -As part of the Data Analytics Framework for [XDMoD](https://open.xdmod.org), this Python package provides API access to the data warehouse of instances of Open XDMoD. +As part of the Data Analytics Framework for [XDMoD](https://open.xdmod.org), +this Python package provides API access to the data warehouse of instances of +Open XDMoD. -This documentation is for **v1.x.y (development branch)**. For documentation of other versions: +This documentation is for **v1.x.y (development branch)**. For documentation of +other versions: - [v1.0.2](https://github.com/ubccr/xdmod-data/tree/v1.0.2?tab=readme-ov-file#xdmod-data) - [v1.0.1](https://github.com/ubccr/xdmod-data/tree/v1.0.1?tab=readme-ov-file#xdmod-data) @@ -12,22 +15,35 @@ The package can be installed from PyPI via `pip install xdmod-data`. Existing installations can be upgraded via `pip install --upgrade xdmod-data`. -The package has dependencies on [NumPy](https://pypi.org/project/numpy/), [Pandas](https://pypi.org/project/pandas/), [Plotly](https://pypi.org/project/plotly/), and [Requests](https://pypi.org/project/requests/). +The package has dependencies on [NumPy](https://pypi.org/project/numpy/), +[Pandas](https://pypi.org/project/pandas/), +[Plotly](https://pypi.org/project/plotly/), +[Python-dotenv](https://pypi.org/project/python-dotenv/), and +[Requests](https://pypi.org/project/requests/). -Example usage is documented through Jupyter notebooks in the [xdmod-notebooks](https://github.com/ubccr/xdmod-notebooks) repository. +Example usage is documented through Jupyter notebooks in the +[xdmod-notebooks](https://github.com/ubccr/xdmod-notebooks) repository. ## Compatibility with Open XDMoD -Specific versions of this package are compatible with specific versions of Open XDMoD as indicated in the table below. +Specific versions of this package are compatible with specific versions of Open +XDMoD as indicated in the table below. | `xdmod-data` version | Open XDMoD versions | | -------------------- | ------------------- | | 1.0.2, 1.0.1 | 11.0.x, 10.5.x | | 1.0.0 | 10.5.x | -## API Token Access -Use of the Data Analytics Framework requires an API token. To obtain an API token, follow the steps below to obtain an API token from the XDMoD portal. +## API Token or JupyterHub Access +Use of the Data Analytics Framework requires an API token, unless you are +running in an XDMoD-hosted JupyterHub such as the one provided by ACCESS XDMoD +(https://xdmod.access-ci.org/jupyter-hub), in which case authentication is +handled automatically. -1. First, if you are not already signed in to the portal, sign in in the top-left corner: +If you need to obtain an API token, you can follow the steps below to obtain +one from the XDMoD portal. + +1. First, if you are not already signed in to the portal, sign in in the + top-left corner: ![Screenshot of "Sign In" button](https://raw.githubusercontent.com/ubccr/xdmod-data/main/docs/images/api-token/sign-in.jpg) @@ -49,17 +65,21 @@ Use of the Data Analytics Framework requires an API token. To obtain an API toke ![Screenshot of "Generate API Token" button](https://raw.githubusercontent.com/ubccr/xdmod-data/main/docs/images/api-token/generate.jpg) -1. Copy the token to your clipboard. Make sure to paste it somewhere secure for saving, as you will not be able to see the token again once you close the window: +1. Copy the token to your clipboard. Make sure to paste it somewhere secure for + saving, as you will not be able to see the token again once you close the + window: ![Screenshot of "Copy API Token to Clipboard" button](https://raw.githubusercontent.com/ubccr/xdmod-data/main/docs/images/api-token/copy.jpg) **Note:** If you lose your token, simply delete it and generate a new one. ## Feedback / Feature Requests -We welcome your feedback and feature requests for the Data Analytics Framework for XDMoD via email: ccr-xdmod-help@buffalo.edu. +We welcome your feedback and feature requests for the Data Analytics Framework +for XDMoD via email: ccr-xdmod-help@buffalo.edu. ## Support -For support, please see [this page](https://open.xdmod.org/support.html). If you email for support, please include the following: +For support, please see [this page](https://open.xdmod.org/support.html). If +you email for support, please include the following: * `xdmod-data` version number, obtained by running this Python code: ``` from xdmod_data import __version__ @@ -70,15 +90,24 @@ For support, please see [this page](https://open.xdmod.org/support.html). If you * Detailed steps to reproduce the problem. ## License -`xdmod-data` is released under the GNU Lesser General Public License ("LGPL") Version 3.0. See the [LICENSE](LICENSE) file for details. +`xdmod-data` is released under the GNU Lesser General Public License ("LGPL") +Version 3.0. See the [LICENSE](LICENSE) file for details. ## References -When referencing the Data Analytics Framework for XDMoD, please cite the following publication: +When referencing the Data Analytics Framework for XDMoD, please cite the +following publication: -> Weeden, A., White, J.P., DeLeon, R.L., Rathsam, R., Simakov, N.A., Saeli, C., and Furlani, T.R. The Data Analytics Framework for XDMoD. _SN COMPUT. SCI._ 5, 462 (2024). https://doi.org/10.1007/s42979-024-02789-2 +> Weeden, A., White, J.P., DeLeon, R.L., Rathsam, R., Simakov, N.A., Saeli, C., + and Furlani, T.R. The Data Analytics Framework for XDMoD. _SN COMPUT. SCI._ + 5, 462 (2024). https://doi.org/10.1007/s42979-024-02789-2 When referencing XDMoD, please cite the following publication: -> Jeffrey T. Palmer, Steven M. Gallo, Thomas R. Furlani, Matthew D. Jones, Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia D. Cornelius, James C. Browne, William L. Barth, Richard T. Evans, "Open XDMoD: A Tool for the Comprehensive Management of High-Performance Computing Resources", *Computing in Science & Engineering*, Vol 17, Issue 4, 2015, pp. 52-62. DOI:[10.1109/MCSE.2015.68](https://doi.org/10.1109/MCSE.2015.68) - +> Jeffrey T. Palmer, Steven M. Gallo, Thomas R. Furlani, Matthew D. Jones, + Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette + Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia D. Cornelius, + James C. Browne, William L. Barth, Richard T. Evans, "Open XDMoD: A Tool for + the Comprehensive Management of High-Performance Computing Resources", + *Computing in Science & Engineering*, Vol 17, Issue 4, 2015, pp. 52-62. + DOI:[10.1109/MCSE.2015.68](https://doi.org/10.1109/MCSE.2015.68) diff --git a/setup.cfg b/setup.cfg index f87aef0f..d3481479 100644 --- a/setup.cfg +++ b/setup.cfg @@ -17,4 +17,5 @@ install_requires = numpy >= 1.23.0 pandas >= 1.5.0 plotly >= 5.8.0 + python-dotenv >= 1.0.0 requests >= 2.19.0 diff --git a/tests/integration/test_datawarehouse_integration.py b/tests/integration/test_datawarehouse_integration.py index abeaf8fa..758c57f5 100644 --- a/tests/integration/test_datawarehouse_integration.py +++ b/tests/integration/test_datawarehouse_integration.py @@ -65,13 +65,13 @@ 'field': (INVALID_STR, r'Field .* not found'), } -key_error_test_ids = [] +param_key_error_test_ids = [] duration_test_ids = [] start_end_test_ids = [] type_error_test_ids = [] default_valid_params = {} -key_error_test_params = [] +param_key_error_test_params = [] date_malformed_test_params = [] type_error_test_params = [] value_error_test_methods = [] @@ -83,9 +83,9 @@ type_error_test_ids += [method + ':' + param] type_error_test_params += [(method, param)] if param in KEY_ERROR_TEST_VALUES_AND_MATCHES: - key_error_test_ids += [method + ':' + param] + param_key_error_test_ids += [method + ':' + param] (value, match) = KEY_ERROR_TEST_VALUES_AND_MATCHES[param] - key_error_test_params += [(method, {param: value}, match)] + param_key_error_test_params += [(method, {param: value}, match)] if param == 'duration': duration_test_ids += [method] start_end_test_ids += [ @@ -107,9 +107,13 @@ value_error_test_methods += [method] if 'filters' in METHOD_PARAMS[method]: for param in ('filter_key', 'filter_value'): - key_error_test_ids += [method + ':' + param] + param_key_error_test_ids += [method + ':' + param] (value, match) = KEY_ERROR_TEST_VALUES_AND_MATCHES[param] - key_error_test_params += [(method, {'filters': value}, match)] + param_key_error_test_params += [( + method, + {'filters': value}, + match, + )] load_dotenv(Path(os.path.expanduser(TOKEN_PATH)), override=True) @@ -130,6 +134,19 @@ def dw_methods_outside_runtime_context(): return __get_dw_methods(dw) +@pytest.fixture(scope='module') +def dw_methods_no_xdmod_api_token(): + token = os.environ['XDMOD_API_TOKEN'] + del os.environ['XDMOD_API_TOKEN'] + with open(Path(os.path.expanduser('~/.xdmod-jwt.env')), 'w') as jwt_file: + jwt_file.write('XDMOD_JWT=' + INVALID_STR + '\n') + dw = DataWarehouse(VALID_XDMOD_HOST) + os.environ['XDMOD_API_TOKEN'] = token + with dw: + os.remove(Path(os.path.expanduser('~/.xdmod-jwt.env'))) + yield __get_dw_methods(dw) + + def __get_dw_methods(dw): return { 'get_data': dw.get_data, @@ -153,12 +170,32 @@ def __test_exception(dw_methods, method, additional_params, error, match): __run_method(dw_methods, method, additional_params) +@pytest.mark.parametrize( + 'method', + list(METHOD_PARAMS.keys()), +) +def test_token_KeyError(dw_methods_no_xdmod_api_token, method): + __test_exception( + dw_methods_no_xdmod_api_token, + method, + {}, + KeyError, + ( + 'If running in JupyterHub connected with XDMoD, this is likely an' + + ' error with the JupyterHub. Otherwise, make sure the' + + ' `XDMOD_API_TOKEN` environment variable is set before the' + + ' `DataWarehouse` is constructed; it should be set to a valid' + + ' API token obtained from the XDMoD web portal.' + ), + ) + + @pytest.mark.parametrize( 'method, params, match', - key_error_test_params, - ids=key_error_test_ids, + param_key_error_test_params, + ids=param_key_error_test_ids, ) -def test_KeyError(dw_methods, method, params, match): +def test_param_KeyError(dw_methods, method, params, match): __test_exception(dw_methods, method, params, KeyError, match) diff --git a/tests/unit/test_datawarehouse_unit.py b/tests/unit/test_datawarehouse_unit.py index 9627e0f1..0111c062 100644 --- a/tests/unit/test_datawarehouse_unit.py +++ b/tests/unit/test_datawarehouse_unit.py @@ -25,17 +25,6 @@ def test___init___TypeError_xdmod_host(): DataWarehouse(2) -def test___init___KeyError(): - token = os.environ['XDMOD_API_TOKEN'] - del os.environ['XDMOD_API_TOKEN'] - with pytest.raises( - KeyError, - match='`XDMOD_API_TOKEN` environment variable has not been set.', - ): - DataWarehouse(VALID_XDMOD_HOST) - os.environ['XDMOD_API_TOKEN'] = token - - def test___enter___RuntimeError_xdmod_host_malformed(): with pytest.raises( ( @@ -44,9 +33,8 @@ def test___enter___RuntimeError_xdmod_host_malformed(): ), match=( r'(Invalid URL \'.*\': No host supplied|' - + r'Invalid URL \'https:\?Bearer=' + INVALID_STR + "': " - + r'No schema supplied. Perhaps you meant http://https:\?Bearer=' - + INVALID_STR + r'\?)' + + r'Invalid URL \'https:\': No schema supplied.' + + r' Perhaps you meant http://https:\?)' ), ): with DataWarehouse('https://'): # pragma: no cover @@ -73,8 +61,13 @@ def test___enter___RuntimeError_xdmod_host_unsupported_protocol(): def test___enter___RuntimeError_401(): with pytest.raises( RuntimeError, - match='Error 401: Make sure XDMOD_API_TOKEN is set' - + ' to a valid API token.', + match=( + 'Error 401: If running in JupyterHub connected with XDMoD, this' + + ' is likely an error with the JupyterHub. Otherwise, make sure' + + ' the `XDMOD_API_TOKEN` environment variable is set before the' + + ' `DataWarehouse` is constructed; it should be set to a valid' + + ' API token obtained from the XDMoD web portal.' + ), ): with DataWarehouse(VALID_XDMOD_HOST) as dw: dw.describe_realms() diff --git a/xdmod_data/_http_requester.py b/xdmod_data/_http_requester.py index 1fd8f99f..a65e3a44 100644 --- a/xdmod_data/_http_requester.py +++ b/xdmod_data/_http_requester.py @@ -1,5 +1,7 @@ +from dotenv import dotenv_values import json import os +from pathlib import Path import re import requests from urllib.parse import urlencode @@ -13,14 +15,10 @@ def __init__(self, xdmod_host): _validator._assert_str('xdmod_host', xdmod_host) xdmod_host = re.sub('/+$', '', xdmod_host) self.__xdmod_host = xdmod_host - try: + self.__api_token = None + if 'XDMOD_API_TOKEN' in os.environ: self.__api_token = os.environ['XDMOD_API_TOKEN'] - except KeyError: - raise KeyError( - '`XDMOD_API_TOKEN` environment variable has not been set.', - ) from None self.__headers = { - 'Authorization': 'Bearer ' + self.__api_token, 'User-Agent': __title__ + ' Python v' + __version__, } self.__requests_session = None @@ -124,17 +122,37 @@ def __assert_connection_to_xdmod_host(self): def __request(self, path='', post_fields=None, stream=False): _validator._assert_runtime_context(self.__in_runtime_context) url = self.__xdmod_host + path + token_error_msg = ( + 'If running in JupyterHub connected with XDMoD, this is likely an' + + ' error with the JupyterHub. Otherwise, make sure the' + + ' `XDMOD_API_TOKEN` environment variable is set before the' + + ' `DataWarehouse` is constructed; it should be set to a valid' + + ' API token obtained from the XDMoD web portal.' + ) + if self.__api_token is not None: + token = self.__api_token + else: + try: + values = dotenv_values( + Path(os.path.expanduser('~/.xdmod-jwt.env')), + ) + token = values['XDMOD_JWT'] + except KeyError: + raise KeyError(token_error_msg) from None + headers = { + **self.__headers, + **{ + 'Authorization': 'Bearer ' + token, + }, + } if post_fields: - post_fields['Bearer'] = self.__api_token response = self.__requests_session.post( url, - headers=self.__headers, + headers=headers, data=post_fields, ) else: - url += '&' if '?' in url else '?' - url += 'Bearer=' + self.__api_token - response = self.__requests_session.get(url, headers=self.__headers) + response = self.__requests_session.get(url, headers=headers) if response.status_code != 200: msg = '' try: @@ -144,7 +162,7 @@ def __request(self, path='', post_fields=None, stream=False): pass if response.status_code == 401: msg = ( - ': Make sure XDMOD_API_TOKEN is set to a valid API token.' + ': ' + token_error_msg ) raise RuntimeError( 'Error ' + str(response.status_code) + msg, diff --git a/xdmod_data/warehouse.py b/xdmod_data/warehouse.py index c1c56945..363aaf25 100644 --- a/xdmod_data/warehouse.py +++ b/xdmod_data/warehouse.py @@ -15,6 +15,12 @@ class DataWarehouse: >>> with DataWarehouse('https://xdmod.access-ci.org') as dw: ... dw.get_data() + If running in a JupyterHub connected to XDMoD, authentication should + happen automatically. Otherwise, make sure the `XDMOD_API_TOKEN` + environment variable is set before the `DataWarehouse` is + constructed; it should be set to a valid API token obtained from + the XDMoD web portal. + Parameters ---------- xdmod_host : str @@ -22,8 +28,6 @@ class DataWarehouse: Raises ------ - KeyError - If the `XDMOD_API_TOKEN` environment variable has not been set. RuntimeError If a connection cannot be made to the XDMoD server specified by `xdmod_host`. @@ -112,8 +116,9 @@ def get_data( Raises ------ KeyError - If any of the parameters have invalid values. Valid realms - come from `describe_realms()`, valid metrics come from + If an authentication token could not be loaded or if any of the + parameters have invalid values. Valid realms come from + `describe_realms()`, valid metrics come from `describe_metrics()`, valid dimensions and filter keys come from `describe_dimensions()`, valid filter values come from `get_filter_values()`, valid durations come from @@ -182,8 +187,9 @@ def get_raw_data( Raises ------ KeyError - If any of the parameters have invalid values. Valid durations - come from `get_durations()`, valid realms come from + If an authentication token could not be loaded or if any of the + parameters have invalid values. Valid durations come from + `get_durations()`, valid realms come from `describe_raw_realms()`, valid filters keys come from `describe_dimensions()`, valid filter values come from `get_filter_values()`, and valid fields come from @@ -215,6 +221,8 @@ def describe_realms(self): Raises ------ + KeyError + If an authentication token could not be loaded. RuntimeError If this method is called outside the runtime context. """ @@ -243,7 +251,8 @@ def describe_metrics(self, realm): Raises ------ KeyError - If `realm` is not one of the values from `describe_realms()`. + If an authentication token could not be loaded or if `realm` is + not one of the values from `describe_realms()`. RuntimeError If this method is called outside the runtime context. TypeError @@ -270,7 +279,8 @@ def describe_dimensions(self, realm): Raises ------ KeyError - If `realm` is not one of the values from `describe_realms()`. + If an authentication token could not be loaded or if `realm` is + not one of the values from `describe_realms()`. RuntimeError If this method is called outside the runtime context. TypeError @@ -300,9 +310,9 @@ def get_filter_values(self, realm, dimension): Raises ------ KeyError - If `realm` is not one of the values from `describe_realms()` or - `dimension` is not one of the IDs or labels from - `describe_dimensions()`. + If an authentication token could not be loaded or if `realm` is + not one of the values from `describe_realms()` or `dimension` is + not one of the IDs or labels from `describe_dimensions()`. RuntimeError If this method is called outside the runtime context. TypeError @@ -355,6 +365,8 @@ def describe_raw_realms(self): Raises ------ + KeyError + If an authentication token could not be loaded. RuntimeError If this method is called outside the runtime context. """ @@ -383,8 +395,8 @@ def describe_raw_fields(self, realm): Raises ------ KeyError - If `realm` is not one of the values from - `describe_raw_realms()`. + If an authentication token could not be loaded or if `realm` is + not one of the values from `describe_raw_realms()`. RuntimeError If this method is called outside the runtime context. TypeError