Add NSRDB GOES v4 to iotools #2378

williamhobbs · 2025-02-06T18:54:08Z

Closes Add NSRDB GOES V4 #2326
I am familiar with the contributing guidelines
Tests added
Updates entries in docs/sphinx/source/reference for API changes.
Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
Pull request is nearly complete and ready for detailed review.
Maintainer: Appropriate GitHub Labels (including remote-data) and Milestone are assigned to the Pull Request and linked Issue.

williamhobbs · 2025-02-06T20:34:16Z

What's the best practice for lines like this one that end up too long? Do I simply add # noqa or # noqa: F401? (I don't know what F401 is for, but I've seen it used here...)

GOES_URL = NSRDB_API_BASE + "/api/nsrdb/v2/solar/nsrdb-GOES-aggregated-v4-0-0-download.csv"

https://github.com/williamhobbs/pvlib-python/blob/4af81a76b4d8bb86b992b5a87ca81dbec6ecb8d5/pvlib/iotools/goes4.py#L15

mikofski · 2025-02-06T21:57:41Z

You could use parentheses

GOES_URL = NSRDB_API_BASE + ("/api/nsrdb/v2/solar/"
"nsrdb-GOES-aggregated-v4-0-0-download.csv")

or just break the string smaller and add them

API_STUB = "/api/nsrdb/v2/solar/"
ENDPOINT = "nsrdb-GOES-aggregated-v4-0-0-download.csv"
GOES_URL = (NSRDB_API_BASE
    + API_STUB + ENDPOINT)

imo avoid ignoring pep8 warnings or else they become pointless.

williamhobbs · 2025-02-06T22:41:08Z

Thanks, @mikofski!

docs/sphinx/source/whatsnew/v0.11.3.rst

Co-authored-by: Echedey Luis <[email protected]>

williamhobbs · 2025-02-07T00:33:52Z

I'm not 100% sure that I set up the tests in test_goes4.py correctly. I copied what was done in test_psm3.py and manually downloaded and reformatted as needed the test data csv files.

See test_read_goes4_map_variables() in particular. columns_mapped includes variables like 'Ozone' that are part of a manual download but are not addressed in pvlib.

I think it's correct, but I could be misunderstanding the goals of some of the tests.

…-python into nsrdb_goes_v4

williamhobbs · 2025-02-08T02:51:35Z

Ok, made the switch from "goes4" to "psm4".

I haven't done anything with @kandersolar's suggestion for different get_ functions for different endpoints (e.g., get_nsrdb_goes4_aggregated, get_nsrdb_goes4_conus, and get_nsrdb_goes4_tmy). I'll wait for some consensus on that one.

WIP: handful of minor things

…-python into nsrdb_goes_v4

williamhobbs · 2025-02-13T22:12:25Z

Ok, I made an effort to create separate get_ functions in a new branch: https://github.com/williamhobbs/pvlib-python/tree/nsrdb_psm4_dedicated_get_functions.

I'll hold off on merging that with the branch in this PR (https://github.com/williamhobbs/pvlib-python/tree/nsrdb_goes_v4) until someone suggests that I do it - I don't want to make a mess of things too quickly.

AdamRJensen

A few initial comments

AdamRJensen · 2025-03-23T18:35:29Z

pvlib/iotools/psm4.py

+Get NSRDB PSM4
+see
+https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-conus-v4-0-0-download/


Personally, I think the URL should be in the function references and not in the module description.

My suggestion is to have the module comment be just a single one-liner, e.g. "Functions for reading and retrieving data from NSRDB PSM4."

AdamRJensen · 2025-03-23T18:39:00Z

pvlib/iotools/psm4.py

+    'Alpha': 'alpha',
+    'Asymmetry': 'asymmetry',


I don't know what these are and they aren't part of the pvlib standard variables, so it's unusual for us to map them.

AdamRJensen · 2025-03-23T18:40:12Z

pvlib/iotools/psm4.py

+    Retrieve NSRDB PSM4 timeseries weather data from the PSM4 API. The NSRDB
+    is described in [1]_ and the PSM4 API is described in [2]_, [3]_, and
+    [4]_.


Suggested change

Retrieve NSRDB PSM4 timeseries weather data from the PSM4 API. The NSRDB

is described in [1]_ and the PSM4 API is described in [2]_, [3]_, and

[4]_.

Retrieve NSRDB PSM4 timeseries weather data from the PSM4 API.

The NSRDB is described in [1]_ and the PSM4 API is described in [2]_,

[3]_, and [4]_.

AdamRJensen · 2025-03-23T18:42:31Z

pvlib/iotools/psm4.py

+    interval : int, {60, 5, 15, 30}
+        interval size in minutes, must be 5, 15, 30 or 60. Must be 60 for
+        typical year requests (i.e., tmy/tgy/tdy).


I think we should call this time_period which is used by other iotools functions. I believe that @kandersolar regretted naming it interval (which was taken from NSRDBs naming convention).

AdamRJensen · 2025-03-23T18:42:55Z

pvlib/iotools/psm4.py

+    interval : int, {60, 5, 15, 30}
+        interval size in minutes, must be 5, 15, 30 or 60. Must be 60 for
+        typical year requests (i.e., tmy/tgy/tdy).
+    attributes : list of str, optional


Again I think this is better named something like parameters.

AdamRJensen · 2025-03-23T18:49:18Z

pvlib/iotools/psm4.py

+    return parse_psm4(fbuf, map_variables)
+
+
+def parse_psm4(fbuf, map_variables=True):


I personally think we should get rid of the parse functions as it creates a LOT of duplicate documentation and significantly more functions than needed. It's unnecessary clutter and it takes very little to modify read functions to also take filebuffers.

Alternatively, we could consider making the parse functions private.

AdamRJensen · 2025-03-23T18:49:44Z

pvlib/iotools/psm4.py

+    Parse an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
+    is described in [1]_ and the SAM CSV format is described in [2]_.


Suggested change

Parse an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB

is described in [1]_ and the SAM CSV format is described in [2]_.

Parse an NSRDB PSM4 weather file (formatted as SAM CSV).

The NSRDB is described in [1]_ and the SAM CSV format is described in [2]_.

AdamRJensen · 2025-03-23T18:54:42Z

pvlib/iotools/psm4.py

+    Read an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
+    is described in [1]_ and the SAM CSV format is described in [2]_.
+


Suggested change

Read an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB

is described in [1]_ and the SAM CSV format is described in [2]_.

Read an NSRDB PSM4 weather file (formatted as SAM CSV).

The NSRDB is described in [1]_ and the SAM CSV format is described in [2]_.

AdamRJensen · 2025-03-23T18:55:09Z

pvlib/iotools/psm4.py

+
+    Parameters
+    ----------
+    filename: str


I think it should be str or path-like?

AdamRJensen · 2025-03-23T18:57:32Z

pvlib/tests/iotools/test_psm4.py

+@pytest.fixture(scope="module")
+def nrel_api_key():
+    """Supplies pvlib-python's NREL Developer Network API key.
+
+    Azure Pipelines CI utilizes a secret variable set to NREL_API_KEY
+    to mitigate failures associated with using the default key of
+    "DEMO_KEY". A user is capable of using their own key this way if
+    desired however the default key should suffice for testing purposes.
+    """
+    try:
+        demo_key = os.environ["NREL_API_KEY"]
+    except KeyError:
+        warnings.warn(
+            "WARNING: NREL API KEY environment variable not set! "
+            "Using DEMO_KEY instead. Unexpected failures may occur."
+        )
+        demo_key = 'DEMO_KEY'
+    return demo_key


If it's the same key as for the PSM3, then I suggest importing it from that test module (should be possible?) to avoid duplicating code.

AdamRJensen · 2025-03-23T22:16:39Z

Given the unanimous vote and the three weeks passed (see #2326 (comment)), you can go ahead with splitting the functions into individual ones.

Please still consider the comments above 😄

williamhobbs added 5 commits February 6, 2025 10:46

added goes4

485165b

added goes4 to iotools __init__.py

856abc7

change zenith to solar_zenith

026738e

fixed references to psm3

ee9d526

fixed some linter issues

4af81a7

uppdated whatsnew

547bf02

fixed more formating issues

fee3743

echedey-ls reviewed Feb 6, 2025

View reviewed changes

docs/sphinx/source/whatsnew/v0.11.3.rst Outdated Show resolved Hide resolved

williamhobbs and others added 3 commits February 6, 2025 18:00

added tests and test data

2664d03

fixed linter issues

0118317

Update docs/sphinx/source/whatsnew/v0.11.3.rst

2f6817c

Co-authored-by: Echedey Luis <[email protected]>

williamhobbs mentioned this pull request Feb 7, 2025

Add NSRDB GOES V4 #2326

Open

williamhobbs added 3 commits February 7, 2025 20:43

change "goes4" to "psm4"

fe8821d

Merge branch 'nsrdb_goes_v4' of https://github.com/williamhobbs/pvlib…

b3eb5a5

…-python into nsrdb_goes_v4

fix some missed goes4 instances

f5dec0a

wholmgren and others added 10 commits February 12, 2025 09:32

init file goes4 --> psm4

c23ce24

use urljoin

c857047

use str year in test

c216255

remove versionchanged doc warning

3144899

Merge pull request #1 from wholmgren/nsrdb_goes_v4

bd4f760

WIP: handful of minor things

fix function names in whatsnew

30bf1b1

Merge branch 'nsrdb_goes_v4' of https://github.com/williamhobbs/pvlib…

703685a

…-python into nsrdb_goes_v4

another whatsnew fix

d75c5a0

Merge branch 'main' into nsrdb_goes_v4

9c1230c

update docs for API changes

523877b

kandersolar added this to the v0.11.3 milestone Feb 20, 2025

kandersolar added the enhancement label Feb 20, 2025

kandersolar modified the milestones: v0.11.3, v0.11.4 Mar 14, 2025

AdamRJensen reviewed Mar 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add NSRDB GOES v4 to iotools #2378

Add NSRDB GOES v4 to iotools #2378

williamhobbs commented Feb 6, 2025 •

edited by kandersolar

Loading

williamhobbs commented Feb 6, 2025

mikofski commented Feb 6, 2025

williamhobbs commented Feb 6, 2025

williamhobbs commented Feb 7, 2025

williamhobbs commented Feb 8, 2025

williamhobbs commented Feb 13, 2025

AdamRJensen left a comment

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen Mar 23, 2025

AdamRJensen commented Mar 23, 2025

		return parse_psm4(fbuf, map_variables)


		def parse_psm4(fbuf, map_variables=True):

		Parse an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
		is described in [1]_ and the SAM CSV format is described in [2]_.

		Read an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
		is described in [1]_ and the SAM CSV format is described in [2]_.

Add NSRDB GOES v4 to iotools #2378

Are you sure you want to change the base?

Add NSRDB GOES v4 to iotools #2378

Conversation

williamhobbs commented Feb 6, 2025 • edited by kandersolar Loading

williamhobbs commented Feb 6, 2025

mikofski commented Feb 6, 2025

williamhobbs commented Feb 6, 2025

williamhobbs commented Feb 7, 2025

williamhobbs commented Feb 8, 2025

williamhobbs commented Feb 13, 2025

AdamRJensen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdamRJensen commented Mar 23, 2025

williamhobbs commented Feb 6, 2025 •

edited by kandersolar

Loading