Forums

Back

RE: Blacklist by sensor/element or exclude list - erroneous mesonet precip totals

MH
Mike Hardiman, modified 4 Years ago.

Blacklist by sensor/element or exclude list - erroneous mesonet precip totals

Youngling Posts: 16 Join Date: 12/17/13 Recent Posts

Hi,

 

Is there a correct way to simply "blacklist" ALL precipitation elements from a given mesonet station? If I go to the Obs Monitor, with "Observations" selected in the 'Data Selection' drop-down, I can right-click on a variable for a station, and "add to exclude list" -- but only for pcpn1, pcpn6, pcpn12, pcpn24 -- and apparently only if there is non-zero data in that column. (If the column is blank, you only have one choise: "remove non-existent values for pcpn1". 

If I switch to  "Precipitation" in the drop-down, then select "Past 1 hour" (or anything), then "Get Data", slightly different elements show up. For example: "Past 1 hour" will give me "pcpn1h" (not pcpn1). With "Past 2 Days" , the variable is 'pcpn2d'. In this case, if I opt to add the element to the "exclude list", it doesn't stick. Next time I click "refresh obs", the data reappears, though with a different value. 

All I really want to do here is to be able to turn off *all* precipitation obs from specific crummy mesonet stations, or ones clogged up with bird poo, so they won't infest the WxReports/LSR table, and have them stay that way until manually removed from the blacklist. (Or, for that matter, any other poorly-behaving element). 

The only info I can find on this in the User Guide is under ObsMonitor, Adding and Editing Stations. A table reads "Bad Sensor Blacklist (removed in 6.5.0) Blacklisting sensors is not down directly on the table.  Right-click on the element you wish to blacklist and select to blacklist it."  If the functionality was removed in 6.5.0, why, and why does the option to add to "exclude list" still exist? 

If this functionality no longer exists, what is the workaround, besides removing data from the Wx Reports table over and over again?

Further confounding things... I've got at least one station with bad data in IRIS. AP749 (Mountaintop, PA). Shows up in the Obs Monitor "Precipitation" table as 5.56 inches in 24hr ('pcpn24h'), and for some reason just 0.34" in 48hr ('pcpn2d'), while the ob time is only a few mins old. Looking at the raw data on findu.com, and 48hr accumulation on the Weather Hazards Data Viewer (which uses the MesoWest/Synoptic Data API), and the listed obs in MesoWest, the total should only be 4.66 inches. If IRIS is picking up a bad precip total from some RRx SHEF product, I can't really tell. FWIW, this particular station isn't in our HydroDB. There doesn't seem to be a way in IRIS to see where stations' observations are coming in from. The "Only ingest product (6 char...)" field doesn't say how to only ingest from MADIS. 

Thanks for any help you can provide,

-Mike

MS
Mike Sutton, modified 4 Years ago.

RE: Blacklist by sensor/element or exclude list - erroneous mesonet precip totals

Youngling Posts: 61 Join Date: 6/21/13 Recent Posts
Mike,

Sorry for not getting back to you sooner.

I don't believe that there is an easy way to blacklist all precipitation elements for a site.  

If you right click on the location in Obs Monitor and select Get All Obs, it will show the decoder that is used as well as the source file.  

The observation decoder will be updated in v 6.7.5 which should be coming out in the near future.  That may resolve the bad precipitation issue.

Thanks,
ms

On Thu, Sep 2, 2021 at 5:39 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

Hi,

 

Is there a correct way to simply "blacklist" ALL precipitation elements from a given mesonet station? If I go to the Obs Monitor, with "Observations" selected in the 'Data Selection' drop-down, I can right-click on a variable for a station, and "add to exclude list" -- but only for pcpn1, pcpn6, pcpn12, pcpn24 -- and apparently only if there is non-zero data in that column. (If the column is blank, you only have one choise: "remove non-existent values for pcpn1". 

If I switch to  "Precipitation" in the drop-down, then select "Past 1 hour" (or anything), then "Get Data", slightly different elements show up. For example: "Past 1 hour" will give me "pcpn1h" (not pcpn1). With "Past 2 Days" , the variable is 'pcpn2d'. In this case, if I opt to add the element to the "exclude list", it doesn't stick. Next time I click "refresh obs", the data reappears, though with a different value. 

All I really want to do here is to be able to turn off *all* precipitation obs from specific crummy mesonet stations, or ones clogged up with bird poo, so they won't infest the WxReports/LSR table, and have them stay that way until manually removed from the blacklist. (Or, for that matter, any other poorly-behaving element). 

The only info I can find on this in the User Guide is under ObsMonitor, Adding and Editing Stations. A table reads "Bad Sensor Blacklist (removed in 6.5.0) Blacklisting sensors is not down directly on the table.  Right-click on the element you wish to blacklist and select to blacklist it."  If the functionality was removed in 6.5.0, why, and why does the option to add to "exclude list" still exist? 

If this functionality no longer exists, what is the workaround, besides removing data from the Wx Reports table over and over again?

Further confounding things... I've got at least one station with bad data in IRIS. AP749 (Mountaintop, PA). Shows up in the Obs Monitor "Precipitation" table as 5.56 inches in 24hr ('pcpn24h'), and for some reason just 0.34" in 48hr ('pcpn2d'), while the ob time is only a few mins old. Looking at the raw data on findu.com, and 48hr accumulation on the Weather Hazards Data Viewer (which uses the MesoWest/Synoptic Data API), and the listed obs in MesoWest, the total should only be 4.66 inches. If IRIS is picking up a bad precip total from some RRx SHEF product, I can't really tell. FWIW, this particular station isn't in our HydroDB. There doesn't seem to be a way in IRIS to see where stations' observations are coming in from. The "Only ingest product (6 char...)" field doesn't say how to only ingest from MADIS. 

Thanks for any help you can provide,

-Mike


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/17340404 VLab.Notifications@noaa.gov


--
Mike Sutton - KB5ZRC
Information Technology Officer
Weather Forecast Office
4899 Tim Dougherty Drive SE
Grand Rapids, MI  49512-4034
(616) 949-0643 x486
AWIPS2 - Build 20.2.3-28
MD
Matt Davis, modified 4 Years ago.

RE: Blacklist by sensor/element or exclude list - erroneous mesonet precip totals

Youngling Posts: 196 Join Date: 6/14/11 Recent Posts
A lot of the bad precipitation issues I've seen lately are due to SHEF encoding errors, such as sending the accumulator value as precipitation, which gets even worse with longer term sums of precipitation. Blacklisting is fixed in 6.7.5, and I am writing the tests for that version today in order to get it implemented at NCO.

Thanks,
Matt

On Fri, Oct 8, 2021 at 12:46 PM Mike Sutton <VLab.Notifications@noaa.gov> wrote:
Mike,

Sorry for not getting back to you sooner.

I don't believe that there is an easy way to blacklist all precipitation elements for a site.  

If you right click on the location in Obs Monitor and select Get All Obs, it will show the decoder that is used as well as the source file.  

The observation decoder will be updated in v 6.7.5 which should be coming out in the near future.  That may resolve the bad precipitation issue.

Thanks,
ms

On Thu, Sep 2, 2021 at 5:39 AM Mike Hardiman <VLab.Notifications@noaa.gov> wrote:

Hi,

 

Is there a correct way to simply "blacklist" ALL precipitation elements from a given mesonet station? If I go to the Obs Monitor, with "Observations" selected in the 'Data Selection' drop-down, I can right-click on a variable for a station, and "add to exclude list" -- but only for pcpn1, pcpn6, pcpn12, pcpn24 -- and apparently only if there is non-zero data in that column. (If the column is blank, you only have one choise: "remove non-existent values for pcpn1". 

If I switch to  "Precipitation" in the drop-down, then select "Past 1 hour" (or anything), then "Get Data", slightly different elements show up. For example: "Past 1 hour" will give me "pcpn1h" (not pcpn1). With "Past 2 Days" , the variable is 'pcpn2d'. In this case, if I opt to add the element to the "exclude list", it doesn't stick. Next time I click "refresh obs", the data reappears, though with a different value. 

All I really want to do here is to be able to turn off *all* precipitation obs from specific crummy mesonet stations, or ones clogged up with bird poo, so they won't infest the WxReports/LSR table, and have them stay that way until manually removed from the blacklist. (Or, for that matter, any other poorly-behaving element). 

The only info I can find on this in the User Guide is under ObsMonitor, Adding and Editing Stations. A table reads "Bad Sensor Blacklist (removed in 6.5.0) Blacklisting sensors is not down directly on the table.  Right-click on the element you wish to blacklist and select to blacklist it."  If the functionality was removed in 6.5.0, why, and why does the option to add to "exclude list" still exist? 

If this functionality no longer exists, what is the workaround, besides removing data from the Wx Reports table over and over again?

Further confounding things... I've got at least one station with bad data in IRIS. AP749 (Mountaintop, PA). Shows up in the Obs Monitor "Precipitation" table as 5.56 inches in 24hr ('pcpn24h'), and for some reason just 0.34" in 48hr ('pcpn2d'), while the ob time is only a few mins old. Looking at the raw data on findu.com, and 48hr accumulation on the Weather Hazards Data Viewer (which uses the MesoWest/Synoptic Data API), and the listed obs in MesoWest, the total should only be 4.66 inches. If IRIS is picking up a bad precip total from some RRx SHEF product, I can't really tell. FWIW, this particular station isn't in our HydroDB. There doesn't seem to be a way in IRIS to see where stations' observations are coming in from. The "Only ingest product (6 char...)" field doesn't say how to only ingest from MADIS. 

Thanks for any help you can provide,

-Mike


--
Mike Hardiman IRIS Virtual Lab Forum https://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/17340404 VLab.Notifications@noaa.gov


--
Mike Sutton - KB5ZRC
Information Technology Officer
Weather Forecast Office
4899 Tim Dougherty Drive SE
Grand Rapids, MI  49512-4034
(616) 949-0643 x486
AWIPS2 - Build 20.2.3-28

--
Mike Sutton IRIS Virtual Lab Forum http://vlab.noaa.gov/web/iris/iris-forum/-/message_boards/view_message/17990594 VLab.Notifications@noaa.gov


--
Project Manager for IRIS, iNWS, CAP Handler
and NWSChat Long Term Update
Meteorological Development Lab
Office of Science and Technology Integration
NOAA DSRC 3D131, Boulder, Colorado
(608) 406-0537