I use the Comparisons tool in WSUP Viewer to evaluate NBM and NDFD
performance for select elements in recent events. But unfortunately,
the comparison options are limited to deterministic elements. It would
be great to also provide verification analysis for probabilistic
elements too. Results can be visualized in a number of ways:
(1) Simple Percentile - Obs to compute biases. When
a user selects PQPF, for example, both the value of the selected
percentile and the URMA/RTMA QPE and NDFD (or another version of
NBM) for the same period would plot along with the ability to toggle
the "Delta" option on/off.
Note: I think you may be able to do this in WSUP already but
only for 6-h snowfall.
(2) Assessing where meaningful exceedance thresholds were
realized. Taking it one step further, when the probability
of exceedance for a particular exceedance threshold (e.g., Prob of
MaxT >90F) is plotted, the footprint of the same observed
(forecast) threshold (e.g., the 90F contour) from URMA/RTMA
(NDFD/NBM) is outlined in the comparison pop-up window.
(3) Obs Rank in NBM Percentile Space. It would be
useful to have an option to compare the gridded observations
(forecast) from URMA/RTMA (NFDF/NBM) to the full NBM probabilistic
distribution. For example, loading NBM v4.2 PWind and URMA winds as
the comparison option would compute the obs rank (or deterministic
forecast rank if the compare dataset used is NDFD/NBM) in NBM
percentile space. Selecting URMA as the comparison dataset to
highlight areas of low and high biases in the NBM probabilistic data
could provide insight into the degree of spatial variability and the
magnitude of these biases for an event. Further analysis of the
meteorological setup may help identify synoptic and mesoscale features
that contributed to extreme low or high biases (e.g., snowfall
exceeded the 95th percentile where the ingredients for mesoscale
banding aligned).
Note: We can do analysis like this for snow/rainfall using the
GAZPACHO program, but it could be much more adaptable if hosted
on a cloud platform. See the example for a heavy rainfall event in
the SE U.S. The shading in the first
graphic represents the highest NBM PQPF percentiles that were
exceeded in this event at each grid point. Analysis source: AHPS.
The second
graphic is similar, but the percentile or deterministic NBM
forecast that produced the lowest MAE (i.e., was closest to the
analysis) is plotted.
