Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NSRDB GOES v4 to iotools #2378

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

williamhobbs
Copy link
Contributor

@williamhobbs williamhobbs commented Feb 6, 2025

  • Closes Add NSRDB GOES V4 #2326
  • I am familiar with the contributing guidelines
  • Tests added
  • Updates entries in docs/sphinx/source/reference for API changes.
  • Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
  • New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
  • Pull request is nearly complete and ready for detailed review.
  • Maintainer: Appropriate GitHub Labels (including remote-data) and Milestone are assigned to the Pull Request and linked Issue.

@williamhobbs
Copy link
Contributor Author

What's the best practice for lines like this one that end up too long? Do I simply add # noqa or # noqa: F401? (I don't know what F401 is for, but I've seen it used here...)

GOES_URL = NSRDB_API_BASE + "/api/nsrdb/v2/solar/nsrdb-GOES-aggregated-v4-0-0-download.csv"

https://github.com/williamhobbs/pvlib-python/blob/4af81a76b4d8bb86b992b5a87ca81dbec6ecb8d5/pvlib/iotools/goes4.py#L15

@mikofski
Copy link
Member

mikofski commented Feb 6, 2025

You could use parentheses

GOES_URL = NSRDB_API_BASE + ("/api/nsrdb/v2/solar/"
"nsrdb-GOES-aggregated-v4-0-0-download.csv")

or just break the string smaller and add them

API_STUB = "/api/nsrdb/v2/solar/"
ENDPOINT = "nsrdb-GOES-aggregated-v4-0-0-download.csv"
GOES_URL = (NSRDB_API_BASE
    + API_STUB + ENDPOINT)

imo avoid ignoring pep8 warnings or else they become pointless.

@williamhobbs
Copy link
Contributor Author

Thanks, @mikofski!

@williamhobbs
Copy link
Contributor Author

I'm not 100% sure that I set up the tests in test_goes4.py correctly. I copied what was done in test_psm3.py and manually downloaded and reformatted as needed the test data csv files.

See test_read_goes4_map_variables() in particular. columns_mapped includes variables like 'Ozone' that are part of a manual download but are not addressed in pvlib.

I think it's correct, but I could be misunderstanding the goals of some of the tests.

@williamhobbs
Copy link
Contributor Author

Ok, made the switch from "goes4" to "psm4".

I haven't done anything with @kandersolar's suggestion for different get_ functions for different endpoints (e.g., get_nsrdb_goes4_aggregated, get_nsrdb_goes4_conus, and get_nsrdb_goes4_tmy). I'll wait for some consensus on that one.

@williamhobbs
Copy link
Contributor Author

Ok, I made an effort to create separate get_ functions in a new branch: https://github.com/williamhobbs/pvlib-python/tree/nsrdb_psm4_dedicated_get_functions.

I'll hold off on merging that with the branch in this PR (https://github.com/williamhobbs/pvlib-python/tree/nsrdb_goes_v4) until someone suggests that I do it - I don't want to make a mess of things too quickly.

@kandersolar kandersolar added this to the v0.11.3 milestone Feb 20, 2025
@kandersolar kandersolar modified the milestones: v0.11.3, v0.11.4 Mar 14, 2025
Copy link
Member

@AdamRJensen AdamRJensen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few initial comments

Comment on lines +2 to +4
Get NSRDB PSM4
see
https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-conus-v4-0-0-download/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I think the URL should be in the function references and not in the module description.

My suggestion is to have the module comment be just a single one-liner, e.g. "Functions for reading and retrieving data from NSRDB PSM4."

Comment on lines +44 to +45
'Alpha': 'alpha',
'Asymmetry': 'asymmetry',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what these are and they aren't part of the pvlib standard variables, so it's unusual for us to map them.

Comment on lines +78 to +80
Retrieve NSRDB PSM4 timeseries weather data from the PSM4 API. The NSRDB
is described in [1]_ and the PSM4 API is described in [2]_, [3]_, and
[4]_.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Retrieve NSRDB PSM4 timeseries weather data from the PSM4 API. The NSRDB
is described in [1]_ and the PSM4 API is described in [2]_, [3]_, and
[4]_.
Retrieve NSRDB PSM4 timeseries weather data from the PSM4 API.
The NSRDB is described in [1]_ and the PSM4 API is described in [2]_,
[3]_, and [4]_.

Comment on lines +98 to +100
interval : int, {60, 5, 15, 30}
interval size in minutes, must be 5, 15, 30 or 60. Must be 60 for
typical year requests (i.e., tmy/tgy/tdy).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should call this time_period which is used by other iotools functions. I believe that @kandersolar regretted naming it interval (which was taken from NSRDBs naming convention).

interval : int, {60, 5, 15, 30}
interval size in minutes, must be 5, 15, 30 or 60. Must be 60 for
typical year requests (i.e., tmy/tgy/tdy).
attributes : list of str, optional
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again I think this is better named something like parameters.

return parse_psm4(fbuf, map_variables)


def parse_psm4(fbuf, map_variables=True):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally think we should get rid of the parse functions as it creates a LOT of duplicate documentation and significantly more functions than needed. It's unnecessary clutter and it takes very little to modify read functions to also take filebuffers.

Alternatively, we could consider making the parse functions private.

Comment on lines +223 to +224
Parse an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
is described in [1]_ and the SAM CSV format is described in [2]_.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Parse an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
is described in [1]_ and the SAM CSV format is described in [2]_.
Parse an NSRDB PSM4 weather file (formatted as SAM CSV).
The NSRDB is described in [1]_ and the SAM CSV format is described in [2]_.

Comment on lines +356 to +358
Read an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
is described in [1]_ and the SAM CSV format is described in [2]_.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Read an NSRDB PSM4 weather file (formatted as SAM CSV). The NSRDB
is described in [1]_ and the SAM CSV format is described in [2]_.
Read an NSRDB PSM4 weather file (formatted as SAM CSV).
The NSRDB is described in [1]_ and the SAM CSV format is described in [2]_.


Parameters
----------
filename: str
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be str or path-like?

Comment on lines +29 to +46
@pytest.fixture(scope="module")
def nrel_api_key():
"""Supplies pvlib-python's NREL Developer Network API key.

Azure Pipelines CI utilizes a secret variable set to NREL_API_KEY
to mitigate failures associated with using the default key of
"DEMO_KEY". A user is capable of using their own key this way if
desired however the default key should suffice for testing purposes.
"""
try:
demo_key = os.environ["NREL_API_KEY"]
except KeyError:
warnings.warn(
"WARNING: NREL API KEY environment variable not set! "
"Using DEMO_KEY instead. Unexpected failures may occur."
)
demo_key = 'DEMO_KEY'
return demo_key
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's the same key as for the PSM3, then I suggest importing it from that test module (should be possible?) to avoid duplicating code.

@AdamRJensen
Copy link
Member

Given the unanimous vote and the three weeks passed (see #2326 (comment)), you can go ahead with splitting the functions into individual ones.

Please still consider the comments above 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add NSRDB GOES V4
6 participants