Forums

Back

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

RSS (Opens New Window)(Opens New Window)
MH
Mike Hardiman, modified 3 Years ago.

Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 16 Join Date: 12/17/13 Recent Posts

Sorry for the flurry of requests, but they all sort of came to mind simultaneously after we finally had a decent snow storm in my CWA.

As mentioned in another thread, we often use Gazpacho to make storm summary precip maps, as do several other offices. It's able to pull in NOHRSC data as a "first guess" for snowfall, if available, and then uses the "metadata" section of PNS products from our office, and surrounding offices.

However, we often truncate the PNS products at some minimum precip/snowfall value, and never include "zeroes" (which often are ingested from COOP and CoCoRaHS reports). 

This can mess with the analysis, especially if there is no "first guess" grid available yet, which is often the case.

We could just flood the PNS with all reports, including zeroes, in order to make better maps.  But the table section, which is meant to be human-readable, would become bloated, and presumably someone out there actually reads them. (Personally, I think a text PNS table without an accompanying map is sort of antiquated). 

Another possible solution would be to create a WRKPNS with ALL reports, for various mapping software to read, then the real transmitted PNS with limits. But this just adds unnecessary steps to an already tedious process.

A better solution, and the actual change request: When IRIS (or its new incarnation) generates a PNS, allow the user to set different limits/thresholds for the table section and the Metadata section. This would keep the table shorter and more manageable, and give mapping programs ALL the data to munch on, including zero-points.

Thanks!

-Mike

 

JM
Joseph Moore, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 31 Join Date: 11/1/13 Recent Posts
Mike - I would definitely pass these suggestions up through your local management to your regional office as a suggestion. It's not clear (to me, at least) who will be developing the IRIS successor that will handle the LSR/PNS stuff (since that's being spun out of the IMS application currently being developed) but making sure the appropriate folks at your regional office are aware of this field need would be the best way to go to try to get it included as a feature in the IRIS successor. 

-Joe

-Joe

On Tue, Jan 18, 2022 at 12:19 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

Sorry for the flurry of requests, but they all sort of came to mind simultaneously after we finally had a decent snow storm in my CWA.

As mentioned in another thread, we often use Gazpacho to make storm summary precip maps, as do several other offices. It's able to pull in NOHRSC data as a "first guess" for snowfall, if available, and then uses the "metadata" section of PNS products from our office, and surrounding offices.

However, we often truncate the PNS products at some minimum precip/snowfall value, and never include "zeroes" (which often are ingested from COOP and CoCoRaHS reports). 

This can mess with the analysis, especially if there is no "first guess" grid available yet, which is often the case.

We could just flood the PNS with all reports, including zeroes, in order to make better maps.  But the table section, which is meant to be human-readable, would become bloated, and presumably someone out there actually reads them. (Personally, I think a text PNS table without an accompanying map is sort of antiquated). 

Another possible solution would be to create a WRKPNS with ALL reports, for various mapping software to read, then the real transmitted PNS with limits. But this just adds unnecessary steps to an already tedious process.

A better solution, and the actual change request: When IRIS (or its new incarnation) generates a PNS, allow the user to set different limits/thresholds for the table section and the Metadata section. This would keep the table shorter and more manageable, and give mapping programs ALL the data to munch on, including zero-points.

Thanks!

-Mike

 


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/19984381 VLab.Notifications@noaa.gov


--
Joseph J. Moore
Warning Coordination Meteorologist
NOAA/National Weather Service Duluth, MN
Office: 218-729-6483 x726 | Cell: 267-481-0341 | Operations: 218-729-0653
MH
Mike Hardiman, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 16 Join Date: 12/17/13 Recent Posts

I assumed this forum was monitored by the IRIS developers, who are hopefully involved in the IMS app?

While I'll attempt to push this suggestion up the chain-of-command, my general experience is that this sort of insistence on "proper channels" just leads to good ideas dying. All it takes is one  over-worked/distracted/disinterested/apathetic link in the chain, and you're out of luck! I sort of figured that this forum was a more direct route.

MS
Mike Sutton, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 58 Join Date: 6/21/13 Recent Posts
We will add this as a feature request.

Thanks,
ms

IRIS Support/ITO GRR

On Fri, Jan 28, 2022 at 11:02 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

I assumed this forum was monitored by the IRIS developers, who are hopefully involved in the IMS app?

While I'll attempt to push this suggestion up the chain-of-command, my general experience is that this sort of insistence on "proper channels" just leads to good ideas dying. All it takes is one  over-worked/distracted/disinterested/apathetic link in the chain, and you're out of luck! I sort of figured that this forum was a more direct route.


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/20155171 VLab.Notifications@noaa.gov


--
Mike Sutton - KB5ZRC
Information Technology Officer
Weather Forecast Office
4899 Tim Dougherty Drive SE
Grand Rapids, MI  49512-4034
Google Voice:  (616) 379-9216
Office:  (616) 949-0643 x486
AWIPS2 - Build 20.2.3-43
MH
Mike Hardiman, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 16 Join Date: 12/17/13 Recent Posts

The system works!! :) 

Thanks, Mike. Let me know if you have any follow up questions, etc. I could probably find other end-users interested in this if that helps. 

MD
Matt Davis, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 193 Join Date: 6/14/11 Recent Posts
Hi Everyone,

Recall that we are now exposing the CSV files we use to create the observation maps in IRIS. These do include the zeroes, and may be more beneficial for mapping. An example is attached.

Thanks,
Matt


On Fri, Jan 28, 2022 at 9:12 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

The system works!! :) 

Thanks, Mike. Let me know if you have any follow up questions, etc. I could probably find other end-users interested in this if that helps. 


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/20156387VLab.Notifications@noaa.gov


--
​MDL ​Project Manager ​for IDSS and Dissemination Systems
Meteorological Development Lab​/WIAD​
Office of Science and Technology Integration
NOAA DSRC 3D131, Boulder, Colorado
(608) 406-0537



Attachments:

MH
Mike Hardiman, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 16 Join Date: 12/17/13 Recent Posts

Interesting... but it looks like it's non-QC'd mesonet obs (+COOP and CoCoRaHS), rather than what's in the Weather Reports Table, right? That would leave out manually-entered reports and whatever comes in via E-Spotter/iNWS. Plus, you'd have to remove bad obs again. 

It's still useful, and maybe easier than using the MesoWest/Synoptic Data API, though MW allows you to put in any time frame for accumulation (including a several years long archive). Where does one find the live data? 

Besides the Gazpacho analyzed maps, I'd like to do something like what Salt Lake City does with their PNS products. Interestingly, they don't include a Metadata section in their text PNS, and I assume they are using ECLAIRS or something else to generate the geolocated data for the plots. But really, it'd be ideal if we could do all this (Gazpacho, PNS plots) from the PNS Metadata section to streamline the process in operations. 
https://www.weather.gov/slc/MapPNS (Click on a past product, then select precip or snow, etc)

I'm personally OK with dumping zeroes into the PNS table to achieve that goal, but I assume there are some users of the PNS who'd be annoyed.

-Mike

BG
Barry S Goldsmith, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 2 Join Date: 12/16/13 Recent Posts
Hi everyone,

*On a low soap box*

I'm just a "lurker" on this discussion, but want to digress (a bit) and add that while I really like the Gazpacho map system for automation, can we *finally* do something about the colors used for increasing rain and snowfall amounts?  How and why were yellow to orange to red selected for most of the considerable ranges?  I've understood that this decision may have had something to do with color blindness...but that is solvable by creating an option (radio button or slider) for the 8% of the male population (many fewer among the female population) that needs to consider an alternate scale.  

To my eyes, and many others not taking the time to look at legends, these "warm to hot" colors are best suited for dryness and heat (drought, wildfire threat, etc).  Also, they depart markedly from other rain/snow QPF/QPE type map color scales, which feature the cooler ("wetter") colors until the very top of the scale, when they basically run out of colors.  Those (as well as shadings) tend to make more sense.

https://www.nohrsc.noaa.gov/interactive/html/map.html
https://www.wpc.ncep.noaa.gov/qpf/fill_94qwbg.gif

Thoughts welcome.

*Off a low soap box*

Regards,

Barry



On Fri, Jan 28, 2022 at 11:16 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

Interesting... but it looks like it's non-QC'd mesonet obs (+COOP and CoCoRaHS), rather than what's in the Weather Reports Table, right? That would leave out manually-entered reports and whatever comes in via E-Spotter/iNWS. Plus, you'd have to remove bad obs again. 

It's still useful, and maybe easier than using the MesoWest/Synoptic Data API, though MW allows you to put in any time frame for accumulation (including a several years long archive). Where does one find the live data? 

Besides the Gazpacho analyzed maps, I'd like to do something like what Salt Lake City does with their PNS products. Interestingly, they don't include a Metadata section in their text PNS, and I assume they are using ECLAIRS or something else to generate the geolocated data for the plots. But really, it'd be ideal if we could do all this (Gazpacho, PNS plots) from the PNS Metadata section to streamline the process in operations. 
https://www.weather.gov/slc/MapPNS (Click on a past product, then select precip or snow, etc)

I'm personally OK with dumping zeroes into the PNS table to achieve that goal, but I assume there are some users of the PNS who'd be annoyed.

-Mike


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/20158056 VLab.Notifications@noaa.gov


--
Barry S. Goldsmith | Warning Coordination Meteorologist
National Weather Service Brownsville/Rio Grande Valley, TX
956-572-1492
Forecast Operations:  956-504-1432 ext. 2

 

AT
Alexander Tardy, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 8 Join Date: 12/12/13 Recent Posts
The Graphical PNS is nice, but we already have the LSR viewer for snow (which could plot wind too). The wind gust
looks like the NWS observation page (aka reads the live mesowest and the syonptic data history) which is already on all WFO pages and can plot wind maxes (though likely
harder for average user). That may be the best feature on the SLC page as it also displays the PNS text next to the wind gust plots.

LSR snowfall plotter (text and plot of values on map)

Running color fill and contours on the snowfall is hard, especially since the reporting is all day, 6 am with 3 inches but 4 pm with 8 inches
of snowfall during a storm even though similar location but different observation and time. You can watch this in the upcoming Boston storm.
The national NOHRSC snowfall may be best on that and water.weather.gov for precip. Not sure the best color curves in this day of technology
the user should be able to pick several scales and colors (as federal we are supposed to follow color blind colors too like we now do in DSS color risk tables). 

Maybe the difference is that one office uses mostly PNS and some mostly LSR. We use both plus an RRM for rainfall numbers.



Thanks, Alex


Alex Tardy


Warning Coordination Meteorologist, Manager
Emergency Preparedness and Partner Collaboration
Education and Outreach Coordinator 
Media and Public Information Officer
Cell: 858-442-6016  Office: 858-675-8700
Skywarn Program Manager
NOAA/National Weather Service
11440 W. Bernardo Court, San Diego, CA

Facebook page for NWS San Diego
Twitter @NWSSanDiego
We need precipitation reports! http://cocorahs.org/



On Fri, Jan 28, 2022 at 10:31 AM Barry S Goldsmith <VLab.Notifications@noaa.gov> wrote:
Hi everyone,

*On a low soap box*

I'm just a "lurker" on this discussion, but want to digress (a bit) and add that while I really like the Gazpacho map system for automation, can we *finally* do something about the colors used for increasing rain and snowfall amounts?  How and why were yellow to orange to red selected for most of the considerable ranges?  I've understood that this decision may have had something to do with color blindness...but that is solvable by creating an option (radio button or slider) for the 8% of the male population (many fewer among the female population) that needs to consider an alternate scale.  

To my eyes, and many others not taking the time to look at legends, these "warm to hot" colors are best suited for dryness and heat (drought, wildfire threat, etc).  Also, they depart markedly from other rain/snow QPF/QPE type map color scales, which feature the cooler ("wetter") colors until the very top of the scale, when they basically run out of colors.  Those (as well as shadings) tend to make more sense.

https://www.nohrsc.noaa.gov/interactive/html/map.html
https://www.wpc.ncep.noaa.gov/qpf/fill_94qwbg.gif

Thoughts welcome.

*Off a low soap box*

Regards,

Barry



On Fri, Jan 28, 2022 at 11:16 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

Interesting... but it looks like it's non-QC'd mesonet obs (+COOP and CoCoRaHS), rather than what's in the Weather Reports Table, right? That would leave out manually-entered reports and whatever comes in via E-Spotter/iNWS. Plus, you'd have to remove bad obs again. 

It's still useful, and maybe easier than using the MesoWest/Synoptic Data API, though MW allows you to put in any time frame for accumulation (including a several years long archive). Where does one find the live data? 

Besides the Gazpacho analyzed maps, I'd like to do something like what Salt Lake City does with their PNS products. Interestingly, they don't include a Metadata section in their text PNS, and I assume they are using ECLAIRS or something else to generate the geolocated data for the plots. But really, it'd be ideal if we could do all this (Gazpacho, PNS plots) from the PNS Metadata section to streamline the process in operations. 
https://www.weather.gov/slc/MapPNS (Click on a past product, then select precip or snow, etc)

I'm personally OK with dumping zeroes into the PNS table to achieve that goal, but I assume there are some users of the PNS who'd be annoyed.

-Mike


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/20158056 VLab.Notifications@noaa.gov


--
Barry S. Goldsmith | Warning Coordination Meteorologist
National Weather Service Brownsville/Rio Grande Valley, TX
956-572-1492
Forecast Operations:  956-504-1432 ext. 2

 


--
Barry S Goldsmith IRIS Virtual Lab Forum http://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/20161570 VLab.Notifications@noaa.gov
MH
Mike Hardiman, modified 3 Years ago.

RE: Change Request: PNS - Differential Filtering (Allow "zero" and low-end reports to show up in Metadata even if they're not in the table)

Youngling Posts: 16 Join Date: 12/17/13 Recent Posts

The LSR viewer is OK, but limited. There's no archive feature, and you can only set the time range in one direction (reports XX hours old, etc). There's also still the problem of low-end and "zero" reports, which I assume would be even less desirable transmitted as LSRs. Also, with LSRs you can issue corrections, but no retractions. I'm not sure how the LSR viewer would handle a corrected location in an LSR, and I need to check in realtime to see how it handles continuous updates for the same location. It's good for a quick look, but not for a final summary product. 

With the graphical PNS you can essentially issue a "curated" list/map of QC'd reports, plucking out stale observations, or reports later deemed too high/low, or later found to be mis-located. A final report can include all spotter reports that didn't "go stale" (i.e., are representative of the storm total), CoCoRaHS, COOP, Snotel reports, etc. It can then be archived for inclusion in a final storm summary, and used in the future for other GIS applications.

Leveraging GIS applications and our myriad observations to easily create high quality, QC'd storm summary products seems like a very basic need. "How much rain/snow fell" or "how high were the wind gusts" are very common partner requests, and we have very clunky methods of answering these questions at present. 

-Mike