Web Content Display

Web Content Display (CoastWatch User Forum)

The NOAA CoastWatch Forum provides information in several categories.  The information is relevant to CoastWatch products and software (https://coastwatch.noaa.gov)  and vary in technical details.  For questions and issues not addressed here,  start a thread in the appropriate category or contact the CoastWatch HelpDesk (coastwatch.info@noaa.gov).  Please note our Privacy Policy (https://www.noaa.gov/privacy.html) and that this is a public forum.  All information is voluntarily and anonymous submissions are accepted pending review prior to posting. 

Forums

Back

cwregister2: [WARNING] Unspecified fill value

MS
Michael Soracco, modified 4 Years ago.

cwregister2: [WARNING] Unspecified fill value

Youngling Posts: 40 Join Date: 8/11/16 Recent Posts

Warnings given by CoastWatch command-line utilities indicate unexpected results may occur even though the program may successfully run to completion.  

Warning messages cannot be suppressed.  As an example,  using cwregister on a Level-2 Ocean Color grouped NetCDF dataset that does not have a _FillValue attribute in the l2_flags variable,  the following may be seen:

 

[WARNING] Unspecified fill value for variable geophysical_data/l2_flags
[WARNING] Set fill value before writing data to avoid unexpected results

 

This warning indicates one of the variables (namely,  geophysical_data/l2_flags )  lacks a designated fill value (which is recommended for the NetCDF Library and Climate-Forecast NetCDF compliance).  Using 'ncdump -h' [from the NetCDF library] you can obtain the attributes in a file.  Also,  the contents below are accessible via THREDDS OpenDAP:

 

/geophysical_data/l2_flags: Array of 32 bit Integers [number_of_lines = 0..767][pixels_per_line = 0..3199]

long_name: Level-2 Processing Flags
coordinates: time longitude latitude
grid_mapping: coord_ref
flag_masks: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 2097152, 4194304, 8388608, 16777216, 33554432, 67108864, 134217728, 268435456, 536870912, 1073741824, -2147483648
flag_meanings: ATMFAIL LAND LOWLWCORR HIGLINT HILT HISATZEN COASTZ LANDADJ STRAYLIGHT CLOUD SPARE10 TURBIDW HISOLZEN HITAU LOWLW CHLFAIL NAVWARN ABSAER CLDSHDSTL MAXAERITER MODGLINT CHLWARN ATMWARN ALGICE SEAICE NAVFAIL FILTER ALTCLD FOG FROMSWIR PRODFAIL NEGLRC
_ChunkSizes: 10, 1067

 

The lack of a fill value may create an issue in either the Java NetCDF or HDF libraries use in the utilities.  Typically,  Java NetCDF will write a zero and HDF may write large values in the output file.  It is recommended to set a _FillValue so that when working with data (i.e.  mapping)  empty parts of a map will not be arbitrarily filled.  An example is seen in the image below.  The blue box is an area where the data from the original L2 chlorophyll-a did not touch and the 512x512 chunk written by the HDF library defaulted to a value that conflicts with actual data:

 

The recommended solution is to set a _FillValue in the L2 data [at the time of this post,  the NOAA MSL12 Ocean Color Level-2 contains Fill Values for variables except for l2_flags]:

 

/geophysical_data/chlor_a: Array of 32 bit Reals [number_of_lines = 0..767][pixels_per_line = 0..3199]

long_name: Chlorophyll Concentration, OC3 Algorithm
C_format: %.4g
units: mg m^-3
coordinates: time longitude latitude
grid_mapping: coord_ref
standard_name: mass_concentration_of_chlorophyll_a_in_sea_water
_FillValue: -999.0
valid_min: 0.001
valid_max: 1000.0
reference: O'Reilly J. E. et al. (1998), Ocean color chlorophyll algorithms for SeaWiFS, J. Geophys. Res., 103(C11), 24937-24953
_ChunkSizes: 10, 1067

Attribute names commencing with underscore ('_') are reserved for use by the netCDF library.

 

The attribute can be set by using the NetCDF Operators (External website: http://nco.sourceforge.net/).  The following command will set the _FillValue to the same value as the Navigation Failure (since all values are used in the bit flags, and the fill value would be for where there is no data):  

Command:  ncatted --attribute _FillValue,l2_flags,c,i,33554432 V2020245173219_NPP_NRTNIR_L2.nc

Result:   l2_flags:_FillValue = 33554432 ;

Additional information regarding fill values (from the external Unidata website):

_FillValue

The _FillValue attribute specifies the fill value used to pre-fill disk space allocated to the variable. Such pre-fill occurs unless nofill mode is set using nc_set_fill(). The fill value is returned when reading values that were never written. If _FillValue is defined then it should be scalar and of the same type as the variable. If the variable is packed using scale_factor and add_offset attributes (see below), the _FillValue attribute should have the data type of the packed data.

It is not necessary to define your own _FillValue attribute for a variable if the default fill value for the type of the variable is adequate. However, use of the default fill value for data type byte is not recommended. Note that if you change the value of this attribute, the changed value applies only to subsequent writes; previously written data are not changed.

Generic applications often need to write a value to represent undefined or missing values. The fill value provides an appropriate value for this purpose because it is normally outside the valid range and therefore treated as missing when read by generic applications. It is legal (but not recommended) for the fill value to be within the valid range.

cwregister2 warning message fillvalue netcdf attributes