Possibly Integrate Radar Sources and/or Station Data

[cloneofghosts]

I originally added this as a comment as part of a separate issue but I'm not sure if you saw it or not so I figured I'd just open it as a separate issue. I haven't found a global source of radar or the one that I found was four hours behind which isn't useful at all for the API.

The closest thing I could find to a global API is this from RainViewer might still be useful as it combines all available radar sources into one source. This from EUMETSTAT while not global might still be useful to cover areas outside of North America which aren't covered by RainViewer though it is lagging behind by about 30m.

From looking at both of the sites they don't seem to offer the data in formats that would be particularly easy for you to use/integrate. I know there's a way to convert GeoTiff files into a format that would be easier for you to use but I don't know how well the conversion works.

[alexander0042]

Thanks for opening this issue- radar data is definitely a really important part of this, and I like the idea of centralizing the discussion here,

RainViewer has some something very cool by integrating all the various radar sources, so that's a starting point. Their API seems very quick, so I might drop them a line to see if I could just query it directly, but otherwise, my thought here is to start with the US NOAA dataset (since it's already on AWS), see if I can get that working, and go from there

[cloneofghosts]

Rainviewer is nice because it includes more sources then the NEXTRAD dataset which would help areas outside of the US which currently have no sources of precipitation for current conditions.

They also have a satellite viewer as well but for some reason has no data over part of the world which is a bit odd and takes away from it's usefulness.

[cloneofghosts]

I'm sure this is on the list of things to work on in the new year but just going to pop in and say that there's been a few times where radar data would have been useful (like now as an example).

[bialylis]

Just note that the rain viewer uses unfiltered radar data in US, which means that there are issues with radar showing dust, birds etc as rain. I have contacted them about it but they are saying they don't have plans to change it in the near term. For my project I'm moving to accessing NOAA data directly from https://mrms.ncep.noaa.gov/data/2D/MergedBaseReflectivityQC/ and falling back to RainViewer in other countries.

[cloneofghosts]

Just note that the rain viewer uses unfiltered radar data in US, which means that there are issues with radar showing dust, birds etc as rain.

I've also noticed the same thing in Eastern Ontario & Southern Quebec when I compare their live maps to those from Environment Canada.

Might be worth combining MRMS and Rainviewer for better results?

[pdxvr]

Even if you use QCd data, there's no guarantee that what you are seeing on the radar is what you are seeing on the ground. As someone who lives at the limit of the radar I can tell you that what's happening at 8000-10000 above me is not what I'm looking at out my window.
https://cliffmass.blogspot.com/2013/02/the-deceptive-weather-radar.html
https://cliffmass.blogspot.com/2014/11/the-other-radar-gap-eastern-slopes-of.html

[cloneofghosts]

I am aware that there are inaccuracies with weather radar but unless you have another option to add in precipitation data this might be the best solution.

I know that station data exists but it has its own set of issues but it is only available where people have setup stations which limits its usefulness a bit.

[pdxvr]

aprs.fi gauges?

[cloneofghosts]

I've never heard of that site before and I wasn't exactly sure what I was looking at when I visited the site.

What does that site offer and how would it be useful for the API?

[pdxvr]

It just popped into my head as a possible source, may not be useful.

There are personal weather stations on there where you can query the data through their API. Right side of page -> filter icon -> Station type: Weather. Although I've only pulled current temps from the site, there may be a way to pull the rain data. The density of gauges is probably = population density and may not be dense enough for purposes, but I feel like radar is honestly so unreliable that anything would be better.

[cloneofghosts]

I know that this exists for station data https://madis.noaa.gov/ but I'm not sure how easy to add it would be.

I'm surprised that you say radar data is unreliable since I'm fairly certain most sites use radar and station data in their current conditions.

Out of curiosity where are you located? I haven't found radar data to be that bad where I am but it probably varies by location

[pdxvr]

Well yes I live in a very hard to predict weather place, Oregon. 25 years in Portland where maybe 30% of the time echoes either meant it was raining or not. 70% of the time reliable. Have to check the traffic cams to be sure. I think it might be related to proximity to the ocean and lower clouds producing rain compared to other places.

Now I live 50 miles from the closest radar site and it’s about 50% accurate. So to qualify what I mean, if you don’t live near a radar site, the echoes are coming from so high up I’m not sure it’s useful.

I could try to run an analysis to see what % of the population lives < 50 miles from radar sites.

My experience on the Midwest and east coast says it’s more reliable there.

[cloneofghosts]

I was thinking that if radar data was integrated maybe an option to not use that data in the API response or if station data is integrated maybe have an option to prioritize that over radar data?

I think a combination of radar/station data would help especially outside of the CONUS domain since there is no precipitation data available and it would help current conditions there. There's been numerous times where HRRR has completely missed precipitation and having some sort of other data source would help with that.

[pdxvr]

I should probably just keep out of it. I live in a place where the weather changes every 10 minutes. I think I probably have some biases about weather accuracy.

[cloneofghosts]

It's always good to hear different perspectives on things and your feedback is very valuable.

As I stated in my previous post the solution here if radar data was incorporated there could be an option to not include it in the API. That way those people who would rather not use radar data have the option to not include it.

This seems like a fair solution to me and probably wouldn't be that difficult to implement.

[alexander0042]

I'm generally of the same mind here- more data sources ingested is always better, but give people the option to select the ones that are most relevant for them. Radar definitely has a time and a place, but it's no substitute for actual observations. Dark Sky defaulted to including it where it was available, and probably had some sort of range limit to the stations.

This is the source I'm planning on using for current conditions: https://registry.opendata.aws/noaa-rtma/. It's not radar based, but integrated a ton of data into a single gridded product. The big perk over the current system is that this updates every 15 minutes (except for precipitation, which is hourly), so will reduce the API lag considerably.

[cloneofghosts]

I know the upcoming 3D version has precipitation included but I wasn't aware that the regular one did as well. I think that having station data with radar as a fallback if there is limited station data would probably work. Maybe in the roadmap moving investigate radar data to be the last item and changing it to include station data might be the thing here?

I'm mainly thinking of outside of the CONUS region where rapidly updating models are limited and would help with current conditions more.

EDIT: I was thinking about this and the option to include/exclude sources would probably want to keep GFS/GEFS always included since HRRR doesn't have a long enough forecast period to give a 7 day forecast (or does the extended one go out that far?). Maybe an option to prioritize station data over radar would also work here? I know in the winter depending on location won't have precipRate data unless the precipitation is not snow.

[cloneofghosts]

The 3DRTMA has been delayed to 2024 which is a bit unfortunate but from what I see it should have pretty much the same values as HRRR. It'll also have a 15m version covering all of NA and I did see a PDF mention the possibility of a global RTMA but it's likely many years away. https://epic.noaa.gov/wp-content/uploads/2022/07/3.-SRW-Rancic.pdf

[cloneofghosts]

Something I noticed with the on/off snow we were having yesterday is that while for the most part Pirate Weather showed no snow but when you look at around 01/02Z this morning it shows precipitation in the area.

I know reflectivity isn't used in PW but I'm wondering if it's a case of the precipitation rate being too low to display it (see #29) or if it just showed nothing which is why it didn't show any precipitation. @alexander0042

Wasn't totally sure where to add this comment but I feel like this spot is best?

[cloneofghosts]

@alexander0042 I noticed today that the API has been reporting cloudy conditions for the most part this afternoon apart from some rare occasions where it did report rain whereas the airport has been reporting rain since noon. (I have the one on my computer update every 20m and its only shown rain twice so far)

Usually the API is pretty good about showing precipitation so not sure if there's something with HRRR where its not picking up the rain or something else is going on?

This is what SpotWX shows for my location if interested:

I know this isn't high on the priority list so maybe once open-sourced this might be improved? There aren't any stations around me which report precipitation so including that won't be super helpful here.