Thanks for the extra details Matt.

So maybe Synoptic_N/DAVIS is having some tech issues that opened up some erroneous data floodgate?

So, since NCEP data comes from MADIS, if MADIS has an obs network of robust data in it already, say LCRA here in TX, that data is still flowing and being used in RTMA/URMA?

I ask, as in your original email you stated "The attached file contains a non-exhaustive list of the providers that are now missing in the RTMA/URMA systems." and on those lists were mentions of LCRA, etc. for networks that seem robust and critical to RTMA/URMA.

On Fri, Sep 15, 2023 at 2:29 PM VLab Notifications <VLab.Notifications@noaa.gov> wrote:

Hi Jack,

The data in the disabled tank should not be duplicated in any of the other tanks.  Note that mesonet data at NCEP comes from MADIS.

The problematic new data is coming from subprovider "DAVIS" under provider "SYNOPTIC_N".

This issue would impact all operational systems that use data from the b255/xx030 tank. The 3D-RTMA/URMA jobs are currently running in the development environment only, without 24/7 operational support, continuing to include b255/xx030 tank data.  In the operational runs, this issue tended to be limited to URMA, suggesting the problematic data might be late-arriving.

Thanks,
Matt

On Thu, Sep 14, 2023 at 4:57 PM VLab Notifications <VLab.Notifications@noaa.gov> wrote:

Good to know this is a high priority item for the group.

Related, I noted that the non-exhaustive list of providers/subproviders you shared has MANY networks that are now missing in RTMA/URMA seem to include rather robust observation networks.  Might it be the case that some of these networks are still being used by RTMA/URMA, with the data coming in to other tanks, or through observations data "collectors/aggregators" like MADIS?

Any idea where these "new observation data" came from, seemingly without warning?

By chance does the in-development 3D RTMA handle these data flows with no issue?

On Thu, Sep 14, 2023 at 3:54 PM VLab Notifications <VLab.Notifications@noaa.gov> wrote:

Hi Jeff,

Although there is no timeline for a potential fix, this is a high-priority task for EMC's obs processing group given the impact on RTMA/URMA.

Thanks,
Matt

On Thu, Sep 14, 2023 at 11:28 AM VLab Notifications <VLab.Notifications@noaa.gov> wrote:

Although there is no timeline known about a potential fix, is this considered a high priority by EMC/NCO?   

JPC

Jeff Craven

Meteorologist in Charge

National Weather Service Forecast Office

Pittsburgh PA (WFO PBZ)

192 Shafer Road

Coraopolis/Moon Township

PA 15108

(412) 262-1591 ext 222 work

(412) 522-5176 mobile/text

@jpcstorm

https://www.weather.gov/pbz/

On Thu, Sep 14, 2023 at 8:16 AM VLab Notifications <VLab.Notifications@noaa.gov> wrote:

Hi All:

Late last week, there was an influx of new observational data that has led to repeat failures of the operational obs processing code at NCEP, that has been impacting RTMA and URMA.  In short, mesonet data arrives at NCEP in various tanks based on the provider.  Several of these providers do not have their own dedicated tanks, so this data flows to a "catch-all" tank, specifically b255/xx030.  The operational code is unable to process the sporadic large volume of data stemming from the new observations.  Over the weekend, a special procedure was used to re-run the failed jobs with this data stream disabled.  Given the repeat issues, an ARFC was implemented on Monday (at 1845Z 09/11/2023) to disable the processing of this tank until a fix can be identified.  At this time, there is no timeline for reversing this emergency fix.  The attached file contains a non-exhaustive list of the providers that are now missing in the RTMA/URMA systems.

Thanks,

Matt

--

Matthew Morris

SAIC at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2038

College Park, MD 20740

301-683-3758

--
Matthew Morris RTMA/URMA Discussion Group Virtual Lab Forum http://vlab.noaa.gov/web/715073/home/-/message_boards/view_message/31942433VLab.Notifications@noaa.gov

--
Jeffrey Craven RTMA/URMA Discussion Group Virtual Lab Forum http://vlab.noaa.gov/web/715073/home/-/message_boards/view_message/31953650VLab.Notifications@noaa.gov

--

Matthew Morris

SAIC at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2038

College Park, MD 20740

301-683-3758

--
Matthew Morris RTMA/URMA Discussion Group Virtual Lab Forum http://vlab.noaa.gov/web/715073/home/-/message_boards/view_message/31955897VLab.Notifications@noaa.gov

--
Jack Settelmaier RTMA/URMA Discussion Group Virtual Lab Forum http://vlab.noaa.gov/web/715073/home/-/message_boards/view_message/31978610VLab.Notifications@noaa.gov