Get UK Target data

[peterdudfield]

We would suggest using the UK PVlive national solar generation as target data. This can be retrieved by API and is fairly easy to use.

Context: At OCF we actually predict GSP level solar generation and use a small ML model to sum it up.

I suggest for this project we start simple, with just the UK national. Also note that the capacity changes, so we need to collect that too, which is possible via API above.

OCF might have a script for collecting this already, and can share.

We should think of a place where we can store the data publicly. Becasue its time series data, its shouldn't be too big

[jcamier]

Potential data source:
https://www.metoffice.gov.uk/services/data/met-office-data-for-reuse/discovery

[peterdudfield]

Potential data source: https://www.metoffice.gov.uk/services/data/met-office-data-for-reuse/discovery

yea this would be good for the NWP gridded data #3. For the target data we should use PVLive - https://www.solar.sheffield.ac.uk/pvlive/, they have an api and python package e.t.c

[jcamier]

Have you guys ever tried this API for PV predictions? I just found it today as I was researching our project...

https://joint-research-centre.ec.europa.eu/photovoltaic-geographical-information-system-pvgis/getting-started-pvgis/api-non-interactive-service_en

[peterdudfield]

We have. script here https://github.com/openclimatefix/uk-pv-backtest/blob/main/scripts/get_pvlive_data.py, that should collect the PVLive data. This might need modifitying so it only collections National Solar generation, i.e gsp_id=0

[jcamier]

This is great. Do you want us to use this library then to get the target data and maybe extend it or use this as a baseline in for a specific utility function in open data PVNet?

[peterdudfield]

Sorry, I'm not sure what you mean?

[jcamier]

There are two approaches we can take with open-data-pvnet:

1. Extend the uk-pv-backtest repository:

• We can enhance the features of the get_pvlive_data.py module or add a new module dedicated to collecting National Solar generation data.
• The enhanced or new module would then be included as a dependency in open-data-pvnet.

2. Develop directly within open-data-pvnet:

• Create a new module within open-data-pvnet itself to handle the collection of National Solar generation data.
• This keeps everything self-contained within open-data-pvnet.

[peterdudfield]

I would go for 2. the uk-pv-bactest repo is what we use internal to generate our backtests and can change quite a lot.
I would pull this script into open-data-pvnet and develop / tidy it up there

[jcamier]

@alirashidAR I added a review comment on your PR.

[peterdudfield]

Thanks @alirashidAR for doing the PR.
Would you like to collect the data too. We can then put it on an public s3 bucket ready for everyone to use

[alirashidAR]

Thanks @alirashidAR for doing the PR. Would you like to collect the data too. We can then put it on an public s3 bucket ready for everyone to use

yes I would like to work on that . Thank you !

[alirashidAR]

@peterdudfield
We would collect this data in the bucket and then create samples, right?
And what should the time frame of the collected data be?

[peterdudfield]

2020 onwards? to be safe

[alirashidAR]

The target data should be stored in which format ?

[peterdudfield]

We tried in the past to stored it as netcdf file.

Just to note it would be good to get the power generation, and also the capacity. The capacity can be got from PVLive too. This changes in time, as more PV systems come online

[sadavaidya]

Hello there,

I’m really interested in contributing to this project and would love to help with this issue of getting UK target data. Could you please confirm if this issue is still open and if there’s any existing work or specific guidelines I should be aware of before getting started?

I understand that @alirashidAR has already worked on a script to download the data in the required format using the Python API. I’d love to know how I can further contribute or build upon that work.

Looking forward to your guidance!

Best regards,
Sadashiv

[MAYANK12SHARMA]

Hi @peterdudfield, is there anything left to do? I’d be happy to do it?

[jcamier]

@alirashidAR I believe you are almost done with this issue. Can you let @MAYANK12SHARMA @sadavaidya if there any assistance you may need?

[MAYANK12SHARMA]

Hii @jcamier can you please take a moment for review the latest PR of mine for the issue #65

[MAYANK12SHARMA]

@alirashidAR I believe you are almost done with this issue. Can you let @MAYANK12SHARMA @sadavaidya if there any assistance you may need?
Yes, @alirashidAR, please let me know if you need any assistance from my side.

[alirashidAR]

Hey @MAYANK12SHARMA @sadavaidya, this issue is almost complete. Thank you, but there isn't much left to work on right now.

[sadavaidya]

Thank you @alirashidAR , for letting us know. I will see if I can contribute to any other issue.