Add ability to authenticate with a JWT loaded from a file.

.circleci/config.yml

@@ -28,7 +28,7 @@ jobs:
               docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade flake8 flake8-commas flake8-quotes'
               docker exec -w /home/circleci/project $python_container bash -c 'python3 -m flake8 . --max-complexity=10 --show-source --exclude __init__.py'
               docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install -e .'
-              docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade python-dotenv pytest coverage'
+              docker exec -w /home/circleci/project $python_container bash -c 'python3 -m pip install --upgrade pytest coverage'
               # The minimum version of each dependency should be tested in the
               # container with the minimum Python version.
               if [ "$python_container" = "python-min" ]; then

CHANGELOG.md

@@ -9,6 +9,7 @@
   ([\#50](https://github.com/ubccr/xdmod-data/pull/50),
   [\#53](https://github.com/ubccr/xdmod-data/pull/53),
   [\#56](https://github.com/ubccr/xdmod-data/pull/56)).
+- Add ability to authenticate with a JWT loaded from a file ([\#54](https://github.com/ubccr/xdmod-data/pull/54)).
 
 ## v1.0.2 (2024-10-31)
 
@@ -22,7 +23,8 @@ It is compatible with Open XDMoD versions 11.0.x and 10.5.x.
 
 ## v1.0.1 (2024-09-27)
 
-This release has bug fixes, performance improvements, and updates for compatibility, tests, and documentation.
+This release has bug fixes, performance improvements, and updates for
+compatibility, tests, and documentation.
 
 It is compatible with Open XDMoD versions 11.0.x and 10.5.x.
 

README.md

@@ -1,7 +1,10 @@
 # xdmod-data
-As part of the Data Analytics Framework for [XDMoD](https://open.xdmod.org), this Python package provides API access to the data warehouse of instances of Open XDMoD.
+As part of the Data Analytics Framework for [XDMoD](https://open.xdmod.org),
+this Python package provides API access to the data warehouse of instances of
+Open XDMoD.
 
-This documentation is for **v1.x.y (development branch)**. For documentation of other versions:
+This documentation is for **v1.x.y (development branch)**. For documentation of
+other versions:
 
 - [v1.0.2](https://github.com/ubccr/xdmod-data/tree/v1.0.2?tab=readme-ov-file#xdmod-data)
 - [v1.0.1](https://github.com/ubccr/xdmod-data/tree/v1.0.1?tab=readme-ov-file#xdmod-data)
@@ -12,22 +15,35 @@ The package can be installed from PyPI via `pip install xdmod-data`.
 
 Existing installations can be upgraded via `pip install --upgrade xdmod-data`.
 
-The package has dependencies on [NumPy](https://pypi.org/project/numpy/), [Pandas](https://pypi.org/project/pandas/), [Plotly](https://pypi.org/project/plotly/), and [Requests](https://pypi.org/project/requests/).
+The package has dependencies on [NumPy](https://pypi.org/project/numpy/),
+[Pandas](https://pypi.org/project/pandas/),
+[Plotly](https://pypi.org/project/plotly/),
+[Python-dotenv](https://pypi.org/project/python-dotenv/), and
+[Requests](https://pypi.org/project/requests/).
 
-Example usage is documented through Jupyter notebooks in the [xdmod-notebooks](https://github.com/ubccr/xdmod-notebooks) repository.
+Example usage is documented through Jupyter notebooks in the
+[xdmod-notebooks](https://github.com/ubccr/xdmod-notebooks) repository.
 
 ## Compatibility with Open XDMoD
-Specific versions of this package are compatible with specific versions of Open XDMoD as indicated in the table below.
+Specific versions of this package are compatible with specific versions of Open
+XDMoD as indicated in the table below.
 
 | `xdmod-data` version | Open XDMoD versions |
 | -------------------- | ------------------- |
 | 1.0.2, 1.0.1         | 11.0.x, 10.5.x      |
 | 1.0.0                | 10.5.x              |
 
-## API Token Access
-Use of the Data Analytics Framework requires an API token. To obtain an API token, follow the steps below to obtain an API token from the XDMoD portal.
+## API Token or JupyterHub Access
+Use of the Data Analytics Framework requires an API token, unless you are
+running in an XDMoD-hosted JupyterHub such as the one provided by ACCESS XDMoD
+(https://xdmod.access-ci.org/jupyter-hub), in which case authentication is
+handled automatically.
 
-1. First, if you are not already signed in to the portal, sign in in the top-left corner:
+If you need to obtain an API token, you can follow the steps below to obtain
+one from the XDMoD portal.
+
+1. First, if you are not already signed in to the portal, sign in in the
+   top-left corner:
 
     ![Screenshot of "Sign In" button](https://raw.githubusercontent.com/ubccr/xdmod-data/main/docs/images/api-token/sign-in.jpg)
 
@@ -49,17 +65,21 @@ Use of the Data Analytics Framework requires an API token. To obtain an API toke
 
     ![Screenshot of "Generate API Token" button](https://raw.githubusercontent.com/ubccr/xdmod-data/main/docs/images/api-token/generate.jpg)
 
-1. Copy the token to your clipboard. Make sure to paste it somewhere secure for saving, as you will not be able to see the token again once you close the window:
+1. Copy the token to your clipboard. Make sure to paste it somewhere secure for
+   saving, as you will not be able to see the token again once you close the
+   window:
 
     ![Screenshot of "Copy API Token to Clipboard" button](https://raw.githubusercontent.com/ubccr/xdmod-data/main/docs/images/api-token/copy.jpg)
 
     **Note:** If you lose your token, simply delete it and generate a new one.
 
 ## Feedback / Feature Requests
-We welcome your feedback and feature requests for the Data Analytics Framework for XDMoD via email: [email protected].
+We welcome your feedback and feature requests for the Data Analytics Framework
+for XDMoD via email: [email protected].
 
 ## Support
-For support, please see [this page](https://open.xdmod.org/support.html). If you email for support, please include the following:
+For support, please see [this page](https://open.xdmod.org/support.html). If
+you email for support, please include the following:
 * `xdmod-data` version number, obtained by running this Python code:
     ```
     from xdmod_data import __version__
@@ -70,15 +90,24 @@ For support, please see [this page](https://open.xdmod.org/support.html). If you
 * Detailed steps to reproduce the problem.
 
 ## License
-`xdmod-data` is released under the GNU Lesser General Public License ("LGPL") Version 3.0. See the [LICENSE](LICENSE) file for details.
+`xdmod-data` is released under the GNU Lesser General Public License ("LGPL")
+Version 3.0. See the [LICENSE](LICENSE) file for details.
 
 ## References
 
-When referencing the Data Analytics Framework for XDMoD, please cite the following publication:
+When referencing the Data Analytics Framework for XDMoD, please cite the
+following publication:
 
-> Weeden, A., White, J.P., DeLeon, R.L., Rathsam, R., Simakov, N.A., Saeli, C., and Furlani, T.R. The Data Analytics Framework for XDMoD. _SN COMPUT. SCI._ 5, 462 (2024). https://doi.org/10.1007/s42979-024-02789-2
+> Weeden, A., White, J.P., DeLeon, R.L., Rathsam, R., Simakov, N.A., Saeli, C.,
+  and Furlani, T.R. The Data Analytics Framework for XDMoD. _SN COMPUT. SCI._
+  5, 462 (2024). https://doi.org/10.1007/s42979-024-02789-2
 
 When referencing XDMoD, please cite the following publication:
 
-> Jeffrey T. Palmer, Steven M. Gallo, Thomas R. Furlani, Matthew D. Jones, Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia D. Cornelius, James C. Browne, William L. Barth, Richard T. Evans, "Open XDMoD: A Tool for the Comprehensive Management of High-Performance Computing Resources", *Computing in Science & Engineering*, Vol 17, Issue 4, 2015, pp. 52-62. DOI:[10.1109/MCSE.2015.68](https://doi.org/10.1109/MCSE.2015.68)
-
+> Jeffrey T. Palmer, Steven M. Gallo, Thomas R. Furlani, Matthew D. Jones,
+  Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette
+  Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia D. Cornelius,
+  James C. Browne, William L. Barth, Richard T. Evans, "Open XDMoD: A Tool for
+  the Comprehensive Management of High-Performance Computing Resources",
+  *Computing in Science & Engineering*, Vol 17, Issue 4, 2015, pp. 52-62.
+  DOI:[10.1109/MCSE.2015.68](https://doi.org/10.1109/MCSE.2015.68)

setup.cfg

@@ -17,4 +17,5 @@ install_requires =
     numpy >= 1.23.0
     pandas >= 1.5.0
     plotly >= 5.8.0
+    python-dotenv >= 1.0.0
     requests >= 2.19.0

tests/integration/test_datawarehouse_integration.py

@@ -65,13 +65,13 @@
     'field': (INVALID_STR, r'Field .* not found'),
 }
 
-key_error_test_ids = []
+param_key_error_test_ids = []
 duration_test_ids = []
 start_end_test_ids = []
 type_error_test_ids = []
 
 default_valid_params = {}
-key_error_test_params = []
+param_key_error_test_params = []
 date_malformed_test_params = []
 type_error_test_params = []
 value_error_test_methods = []
@@ -83,9 +83,9 @@
         type_error_test_ids += [method + ':' + param]
         type_error_test_params += [(method, param)]
         if param in KEY_ERROR_TEST_VALUES_AND_MATCHES:
-            key_error_test_ids += [method + ':' + param]
+            param_key_error_test_ids += [method + ':' + param]
             (value, match) = KEY_ERROR_TEST_VALUES_AND_MATCHES[param]
-            key_error_test_params += [(method, {param: value}, match)]
+            param_key_error_test_params += [(method, {param: value}, match)]
         if param == 'duration':
             duration_test_ids += [method]
             start_end_test_ids += [
@@ -107,9 +107,13 @@
             value_error_test_methods += [method]
     if 'filters' in METHOD_PARAMS[method]:
         for param in ('filter_key', 'filter_value'):
-            key_error_test_ids += [method + ':' + param]
+            param_key_error_test_ids += [method + ':' + param]
             (value, match) = KEY_ERROR_TEST_VALUES_AND_MATCHES[param]
-            key_error_test_params += [(method, {'filters': value}, match)]
+            param_key_error_test_params += [(
+                method,
+                {'filters': value},
+                match,
+            )]
 
 
 load_dotenv(Path(os.path.expanduser(TOKEN_PATH)), override=True)
@@ -130,6 +134,19 @@ def dw_methods_outside_runtime_context():
     return __get_dw_methods(dw)
 
 
+@pytest.fixture(scope='module')
+def dw_methods_no_xdmod_api_token():
+    token = os.environ['XDMOD_API_TOKEN']
+    del os.environ['XDMOD_API_TOKEN']
+    with open(Path(os.path.expanduser('~/.xdmod-jwt.env')), 'w') as jwt_file:
+        jwt_file.write('XDMOD_JWT=' + INVALID_STR + '\n')
+    dw = DataWarehouse(VALID_XDMOD_HOST)
+    os.environ['XDMOD_API_TOKEN'] = token
+    with dw:
+        os.remove(Path(os.path.expanduser('~/.xdmod-jwt.env')))
+        yield __get_dw_methods(dw)
+
+
 def __get_dw_methods(dw):
     return {
         'get_data': dw.get_data,
@@ -153,12 +170,32 @@ def __test_exception(dw_methods, method, additional_params, error, match):
         __run_method(dw_methods, method, additional_params)
 
 
+@pytest.mark.parametrize(
+    'method',
+    list(METHOD_PARAMS.keys()),
+)
+def test_token_KeyError(dw_methods_no_xdmod_api_token, method):
+    __test_exception(
+        dw_methods_no_xdmod_api_token,
+        method,
+        {},
+        KeyError,
+        (
+            'If running in JupyterHub connected with XDMoD, this is likely an'
+            + ' error with the JupyterHub. Otherwise, make sure the'
+            + ' `XDMOD_API_TOKEN` environment variable is set before the'
+            + ' `DataWarehouse` is constructed; it should be set to a valid'
+            + ' API token obtained from the XDMoD web portal.'
+        ),
+    )
+
+
 @pytest.mark.parametrize(
     'method, params, match',
-    key_error_test_params,
-    ids=key_error_test_ids,
+    param_key_error_test_params,
+    ids=param_key_error_test_ids,
 )
-def test_KeyError(dw_methods, method, params, match):
+def test_param_KeyError(dw_methods, method, params, match):
     __test_exception(dw_methods, method, params, KeyError, match)
 
 

tests/unit/test_datawarehouse_unit.py

@@ -25,17 +25,6 @@ def test___init___TypeError_xdmod_host():
         DataWarehouse(2)
 
 
-def test___init___KeyError():
-    token = os.environ['XDMOD_API_TOKEN']
-    del os.environ['XDMOD_API_TOKEN']
-    with pytest.raises(
-        KeyError,
-        match='`XDMOD_API_TOKEN` environment variable has not been set.',
-    ):
-        DataWarehouse(VALID_XDMOD_HOST)
-    os.environ['XDMOD_API_TOKEN'] = token
-
-
 def test___enter___RuntimeError_xdmod_host_malformed():
     with pytest.raises(
         (
@@ -44,9 +33,8 @@ def test___enter___RuntimeError_xdmod_host_malformed():
         ),
         match=(
             r'(Invalid URL \'.*\': No host supplied|'
-            + r'Invalid URL \'https:\?Bearer=' + INVALID_STR + "': "
-            + r'No schema supplied. Perhaps you meant http://https:\?Bearer='
-            + INVALID_STR + r'\?)'
+            + r'Invalid URL \'https:\': No schema supplied.'
+            + r' Perhaps you meant http://https:\?)'
         ),
     ):
         with DataWarehouse('https://'):  # pragma: no cover
@@ -73,8 +61,13 @@ def test___enter___RuntimeError_xdmod_host_unsupported_protocol():
 def test___enter___RuntimeError_401():
     with pytest.raises(
         RuntimeError,
-        match='Error 401: Make sure XDMOD_API_TOKEN is set'
-        + ' to a valid API token.',
+        match=(
+            'Error 401: If running in JupyterHub connected with XDMoD, this'
+            + ' is likely an error with the JupyterHub. Otherwise, make sure'
+            + ' the `XDMOD_API_TOKEN` environment variable is set before the'
+            + ' `DataWarehouse` is constructed; it should be set to a valid'
+            + ' API token obtained from the XDMoD web portal.'
+        ),
     ):
         with DataWarehouse(VALID_XDMOD_HOST) as dw:
             dw.describe_realms()

xdmod_data/_http_requester.py

@@ -1,5 +1,7 @@
+from dotenv import dotenv_values
 import json
 import os
+from pathlib import Path
 import re
 import requests
 from urllib.parse import urlencode
@@ -13,14 +15,10 @@ def __init__(self, xdmod_host):
         _validator._assert_str('xdmod_host', xdmod_host)
         xdmod_host = re.sub('/+$', '', xdmod_host)
         self.__xdmod_host = xdmod_host
-        try:
+        self.__api_token = None
+        if 'XDMOD_API_TOKEN' in os.environ:
             self.__api_token = os.environ['XDMOD_API_TOKEN']
-        except KeyError:
-            raise KeyError(
-                '`XDMOD_API_TOKEN` environment variable has not been set.',
-            ) from None
         self.__headers = {
-            'Authorization': 'Bearer ' + self.__api_token,
             'User-Agent': __title__ + ' Python v' + __version__,
         }
         self.__requests_session = None
@@ -124,17 +122,37 @@ def __assert_connection_to_xdmod_host(self):
     def __request(self, path='', post_fields=None, stream=False):
         _validator._assert_runtime_context(self.__in_runtime_context)
         url = self.__xdmod_host + path
+        token_error_msg = (
+            'If running in JupyterHub connected with XDMoD, this is likely an'
+            + ' error with the JupyterHub. Otherwise, make sure the'
+            + ' `XDMOD_API_TOKEN` environment variable is set before the'
+            + ' `DataWarehouse` is constructed; it should be set to a valid'
+            + ' API token obtained from the XDMoD web portal.'
+        )
+        if self.__api_token is not None:
+            token = self.__api_token
+        else:
+            try:
+                values = dotenv_values(
+                    Path(os.path.expanduser('~/.xdmod-jwt.env')),
+                )
+                token = values['XDMOD_JWT']
+            except KeyError:
+                raise KeyError(token_error_msg) from None
+        headers = {
+            **self.__headers,
+            **{
+                'Authorization': 'Bearer ' + token,
+            },
+        }
         if post_fields:
-            post_fields['Bearer'] = self.__api_token
             response = self.__requests_session.post(
                 url,
-                headers=self.__headers,
+                headers=headers,
                 data=post_fields,
             )
         else:
-            url += '&' if '?' in url else '?'
-            url += 'Bearer=' + self.__api_token
-            response = self.__requests_session.get(url, headers=self.__headers)
+            response = self.__requests_session.get(url, headers=headers)
         if response.status_code != 200:
             msg = ''
             try:
@@ -144,7 +162,7 @@ def __request(self, path='', post_fields=None, stream=False):
                 pass
             if response.status_code == 401:
                 msg = (
-                    ': Make sure XDMOD_API_TOKEN is set to a valid API token.'
+                    ': ' + token_error_msg
                 )
             raise RuntimeError(
                 'Error ' + str(response.status_code) + msg,