diff --git a/getting_started.md b/getting_started.md index 5d15b22..5f42739 100644 --- a/getting_started.md +++ b/getting_started.md @@ -117,7 +117,15 @@ For contributors unfamiliar with these concepts, the [Machine Learning Terms](#m Datasets form the backbone of solar forecasting by providing the historical and real-time data required for model training and evaluation. This project leverages a variety of datasets, including weather, solar generation, and climate data. -For a detailed list of datasets and their descriptions, please refer to the [Datasets Guide](datasets.md). +### Met Office UK Deterministic (UKV) +A NWP dataset used for UK solar forecasting. See [Met Office Dataset Documentation](met_office_dataset.md) for detailed information about: +- Variables and their impact on solar forecasting +- Dataset structure and format +- Data quality considerations +- Access instructions via Hugging Face + +### Other Weather Datasets +For a complete list of available weather datasets and their descriptions, see the [Datasets Guide](datasets.md). --- diff --git a/met_office_dataset.md b/met_office_dataset.md new file mode 100644 index 0000000..b16f5f6 --- /dev/null +++ b/met_office_dataset.md @@ -0,0 +1,155 @@ +# Met Office UK Deterministic (UKV) Dataset + +## Overview +The Met Office UK Deterministic (UKV) dataset provides high-resolution weather forecasts for the UK region. This document details the dataset structure, variables, and their relevance to solar forecasting. + +## Dataset Structure +The dataset uses a Lambert Azimuthal Equal Area projection centered on the UK with: +- Height: 970 pixels +- Width: 1042 pixels +- Grid Resolution: 2km +- Temporal Resolution: 60 minutes +- Forecast Range: 54 hours + +## Variables + +### Cloud Coverage Variables +These variables are critical for predicting solar irradiance attenuation: + +- **high_type_cloud_area_fraction** + - Description: Fraction of high-altitude clouds + - Units: 1 (fraction) + - Impact: High clouds typically have less impact on solar radiation than lower clouds + - Typical Range: 0-1 + +- **medium_type_cloud_area_fraction** + - Description: Fraction of medium-altitude clouds + - Units: 1 (fraction) + - Impact: Moderate impact on solar radiation + - Typical Range: 0-1 + +- **low_type_cloud_area_fraction** + - Description: Fraction of low-altitude clouds + - Units: 1 (fraction) + - Impact: Most significant impact on solar radiation + - Typical Range: 0-1 + +- **cloud_area_fraction** + - Description: Total cloud coverage + - Units: 1 (fraction) + - Impact: Overall indicator of solar radiation reduction + - Typical Range: 0-1 + +### Radiation Flux Variables +Direct measurements of solar radiation components: + +- **surface_downwelling_shortwave_flux_in_air** + - Description: Total downward solar radiation at surface + - Units: W m⁻² + - Impact: Primary predictor for solar PV generation + - Typical Range: 0-1000+ W/m² + - Notes: Includes both direct and diffuse radiation + +- **surface_downwelling_longwave_flux_in_air** + - Description: Thermal radiation from atmosphere + - Units: W m⁻² + - Impact: Affects panel temperature and efficiency + - Typical Range: 200-500 W/m² + +- **surface_downwelling_ultraviolet_flux_in_air** + - Description: UV component of solar radiation + - Units: W m⁻² + - Impact: Can affect panel degradation and specific PV technologies + - Typical Range: 0-100 W/m² + +### Meteorological Variables +Environmental conditions affecting solar panel efficiency: + +- **air_temperature** + - Description: Air temperature at 2m height + - Units: K (Kelvin) + - Impact: Panel efficiency decreases with temperature + - Typical Range: 250-320K + - Note: Convert to Celsius by subtracting 273.15 + +- **wind_speed** + - Description: Wind speed at surface level + - Units: m s⁻¹ + - Impact: Affects panel cooling and efficiency + - Typical Range: 0-30 m/s + +- **wind_from_direction** + - Description: Wind direction at surface level + - Units: degrees + - Impact: Can influence panel temperature and local weather patterns + - Range: 0-360° + - Note: 0° is North, 90° is East + +- **lwe_thickness_of_surface_snow_amount** + - Description: Snow depth in water equivalent + - Units: m + - Impact: Affects ground albedo and potential panel coverage + - Typical Range: 0-1m + +### Coordinate System +The dataset uses Lambert Azimuthal Equal Area projection: + +- **projection_x_coordinate** + - Description: X-axis grid coordinates + - Units: m + - Range: Covers UK extent + +- **projection_y_coordinate** + - Description: Y-axis grid coordinates + - Units: m + - Range: Covers UK extent + +### Time Variables +Temporal information for forecasts: + +- **forecast_period** + - Description: Time offset from reference + - Type: timedelta64[ns] + +- **forecast_reference_time** + - Description: Start time of forecast + - Type: datetime64[ns] + +- **time** + - Description: Valid time for forecast step + - Type: datetime64[ns] + +## Usage in Solar Forecasting + +### Primary Predictors +1. **surface_downwelling_shortwave_flux_in_air**: Direct indicator of solar energy availability +2. **cloud_area_fraction** variables: Key for radiation attenuation +3. **air_temperature**: Critical for panel efficiency calculations + +### Secondary Factors +1. **wind_speed**: Panel cooling effects +2. **snow_amount**: Ground reflectance and coverage +3. **UV flux**: Specific panel technology considerations + +## Data Quality Considerations +- Least significant digit information provided for each variable +- Grid mapping information available in lambert_azimuthal_equal_area variable +- All variables follow CF-1.7 conventions + +## Data Availability and Format +This dataset is hosted on Hugging Face at [openclimatefix/met-office-uk-deterministic-solar](https://huggingface.co/datasets/openclimatefix/met-office-uk-deterministic-solar). + +### File Format +- Files are stored in `.zarr.zip` format +- Each file represents a specific timestamp (e.g., `2023-01-16-00.zarr.zip`) +- Zarr format is optimized for: + - Cloud storage access + - Parallel I/O operations + - Efficient chunked access to large arrays + - Integration with data science tools (xarray, pandas, dask) + +## License +British Crown copyright 2022-2024, the Met Office, licensed under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0/). + +## Citation +Met Office UK Deterministic (UKV)2km on a 2-year rolling archive accessed from [AWS Registry](https://registry.opendata.aws/met-office-uk-deterministic). diff --git a/src/open_data_pvnet/configs/met_office_uk_data_config.yaml b/src/open_data_pvnet/configs/met_office_uk_data_config.yaml index 57b91d6..1e61e75 100644 --- a/src/open_data_pvnet/configs/met_office_uk_data_config.yaml +++ b/src/open_data_pvnet/configs/met_office_uk_data_config.yaml @@ -43,8 +43,8 @@ input_data: - radiation_flux_in_uv_downward_at_surface # Downward UV radiation flux at the surface (W/m²) - wind_speed_at_10m # 10-meter wind speed (m/s) - wind_direction_at_10m # 10-meter wind direction (degrees from north) - nwp_image_size_pixels_height: 12 - nwp_image_size_pixels_width: 12 + nwp_image_size_pixels_height: 970 + nwp_image_size_pixels_width: 1042 nwp_provider: met_office nwp_zarr_path: PLACEHOLDER.zarr time_resolution_minutes: 60