How do I get data?

Now that you've installed the program, you need to download some data. You can do this by manually transfering the data from wherever it is to your machine.

If you are working with NDFD or NDGD data, their URLs are located here. You could manually download the data from there, but the degrib program comes with two ways to automatically access those URLS.

  1. On the "Download" tab of the GUI (Graphical User Interface) program tkdegrib, one can highlight the sector (see NDFD/NDGD page for the definition of a sector), or the specific weather element in the sector that one is interested in, and then click either the "Download by ftp" or the "Download by https" buttons. At that point tkdegrib will contact the web and store the file in /degrib/data/'data set'/'sector', where 'data set' is either NDFD or NDGD, and 'sector' is the abreviated name of the sector.

  2. From a command line or a script (Note in MS-Windows replace /degrib/ with c:\ndfd\degrib):
    • /degrib/bin/tcldegrib web.tcl ndfd midatlan,all
      which should get from the "ndfd" data set all weather elements found in the "midatlan" sector.

    • /degrib/bin/tcldegrib web.tcl ndfd conus,maxt,mint
      which should get from the "ndfd" data set the maxt and mint weather elements found in the "conus" sector.

    • /degrib/bin/tcldegrib web.tcl ndgd conus,ozone01
      which should get from the "ndgd" data set the ozone01 forecast found in the "conus" sector.

What if I can't find a sector that I like?

If you can't find an appropriate NDFD sector, the NWS is providing an experimental web service whose intent is to allow the user to get a custom defined sector in GRIB2. It is currently limited to a maximum of 15 degrees by 15 degrees. 

  1. To use the service from the GUI tkdegrib, go to the "File" menu, and select "Configure". Then click on the "Custom" tab. Enter in the coordinates that you are interested in and press "OK". Now when you highlight the "custom" sector on the "Download tab", it will talk to the web service using your coordinates.

  2. To use the service from the command line, you need to first edit the file "/degrib/bin/ndfd.ini", to use your coordinates. You are looking for the "[Custom_Sector]" section.
    After modifying "/degrib/bin/ndfd.ini", you can talk to the web service using:
    • /degrib/bin/tcldegrib web.tcl ndfd custom,all

What can I do with the data?

View the data:

To get an idea of what the data looks like, on "MS-Windows", you can use "tkdegrib" with "superImageGen" and "htmlmaker" to create images (.png) and a means of browsing those images with "MS-Internet Explorer".

  • In the GUI tkdegrib:
    Go to the "Download" tab, highlight the sector that you want images of, and click "Generate Images". That should cause tkdegrib to convert the selected forecasts to "mosaic (*.tlf)" files (see Appendix for a definition). Then tkdegrib calls superImageGen to read the "mosaic (*.tlf)" files and draw the images. Next htmlmaker is called to create html pages to help view the images, and finally IE is called to browse the html.

    Note 1: Unfortunately, superImageGen is provided as an executable which only works on MS-Windows machines.

  • From the command line:
    Currently there is no way to do this from the command line.

Inventory the data:

To see what messages are inside a GRIB file, you can use degrib to inventory it.

  • In the GUI tkdegrib:
    Click on the "GIS" tab, and browse for your file. Double click on it in the top half, and it should fill out the "message #, a short version of the variable name, a long version of the variable name, the level or surface, the reference time, the valid time, and the difference between the valid time and the reference time" in the bottom.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -I
    where "GRIB file" is replaced with the name of your GRIB file.

    This should print out the message number, which byte it starts on, the GRIB version, the variable name (both short and long forms of it), the level or surface, the reference time, the valid time, and the forecast projection.

Probe the data

To get a better feel for the data, you can use degrib to probe the file at a given lat/lon point, or a set of points in a "point file".

  • In the GUI tkdegrib:
    Currently there is no way to do this from the GUI.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -P -pnt 38,-76
    /degrib/bin/degrib "GRIB file" -P -pntFile "foo.txt"
    where "GRIB file" is replaced with the name of your GRIB file, and "foo.txt" is replaced with a file that contains station ID, lat, lon.

    This should read the file, and extract each message in the file. Then it will either bi-linearly interpolate to the given lat/lon point, or find the nearest neighbor, and output that data to stdout. See the "degrib Man Page" (particularly the "PROBE OPTIONS" section) for more details.

Get Meta data about a message:

To get "meta data" or information describing a GRIB message, you can use degrib to create a .txt file. Amongst the "meta data" will be the maximum value, the minimum value, the number of missing values detected, etc.

  • In the GUI tkdegrib:
    Each time you "convert" a message from a file, the GUI automatically creates a .txt file that contains the "meta data". To convert a file, click on the "GIS" tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press "Recommend" to have tkdegrib "recommend" one. Next choose an "File Type", and press the "Generate file" button at the bottom.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -msg 1
    /degrib/bin/degrib "GRIB file" -C -msg all
    /degrib/bin/degrib "GRIB file" -C -Met -msg 2 -out message2.txt
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, and extract the requested messages. The -Met option (on by default) tells it to create a .txt file with the meta data. To turn off the "meta" file option, use -nMet. See the "degrib Man Page" (particularly the "CONVERT OPTIONS" section) for more details.

Get Diagnostics about a message

To get some diagnostics about the message before it has been parsed by the code, you can create a ".is0" file. This can be useful for diagnosing packing and unpacking routines, but you will need ""A GUIDE TO THE CODE FORM FM 92-IX Ext. GRIB"" to understand the results.

  • In the GUI tkdegrib:
    Currently there is no way to do this from the GUI.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -nMet -IS0 -msg 1
    /degrib/bin/degrib "GRIB file" -C -IS0 -msg all
    where "GRIB file" is replaced with the name of your GRIB file.

Convert a message to ASCII Comma Separated file:

You can use degrib to convert a given message to a .csv file, which is a comma delimited ASCII file that contains a x, y, lat, lon, value line for each cell in the grid. This file can be imported into Excel (limited to the first 65,536 lines), or read by other files to populate a SQL database.

  • In the GUI tkdegrib:
    Click on the "GIS" tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press "Recommend" to have tkdegrib "recommend" one. Next choose "CSV" for "File Type", and press the "Generate .csv file" button at the bottom.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Csv
    /degrib/bin/degrib "GRIB file" -C -msg all -nMet -Csv
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, and extract the requested messages. The -Csv option tells it to create the .csv file. See the "degrib Man Page" (particularly the "CONVERT OPTIONS" section) for more details.

Convert a message to ESRI Shapefile:

You can use degrib to convert a given message to three different types of ESRI Shapefiles. The first type (Point), contains a lat/lon for the center of each cell, along with the value of that cell. The second type (Small Polygon) contains a "small" 5 vertex polygon around the edge, along with the value of each cell. The third type (Large Polygon) contains large polygons that were created by merging all the identical cells from the small polygons.

  • In the GUI tkdegrib:
    Click on the "GIS" tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press "Recommend" to have tkdegrib "recommend" one. Next choose "SHP" for "File Type". Now choose the "Type of .shp file" (either point, small polygon, or large polygon) and press the "Generate .shp file" button at the bottom.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Shp -poly 0
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Shp -poly 1
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Shp -poly small
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Shp -poly 2
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Shp -poly big
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, and extract the requested messages. The -Shp option tells it to create the .shp file. The "-poly 0" says to create a "point" shapefile, the "-poly 1" or "-poly small" says to create the "small polygon" shapefile, while the "-poly 2" or "-poly big" says to create the "large polygon" shapefile. See the "degrib Man Page" (particularly the "CONVERT OPTIONS" section and the "SHP SPECIFIC OPTIONS") for more details.

Convert a message to ESRI Spatial analyst or GrADS File format:

You can use degrib to convert a given message to what I termed to be a ".flt" file. The .flt file consists of NxM floats where N, and M are the dimensions of the grid. The reason for the .flt name was because that (along with a ".hdr" file) was what ESRI Spatial analyst appeared to need. It turned out that "GrADS" used the same format except that in addition to the ".flt" file it required a control file (.ctl).

  • In the GUI tkdegrib:
    Click on the "GIS" tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press "Recommend" to have tkdegrib "recommend" one. Next choose "FLT" for "File Type". Now if you want to create the GrADS ".ctl" file, click on "Create GrADS .ctl file". Then press the "Generate .flt file" button at the bottom.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Flt
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Flt -Interp -GrADS
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Flt -GrADS
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, and extract the requested messages. The -Flt option tells it to create the .flt file. The "-GrADS" option says to create the ".ctl" file. The "-Interp" allows GrADS to work better since it tells degrib to interpolate the grid to a lat/lon coverage grid, which saves GrADS from having to do so. See the "degrib Man Page" (particularly the "CONVERT OPTIONS" section and the "FLT SPECIFIC OPTIONS") for more details.

  • Please see the following for more information on degrib and GrADS
  • Please see the following for more information on FORTRAN or C access to .flt files

Convert a message to NetCDF:

You can use degrib to convert a given message to a NetCDF file (".nc"). For more information about the NetCDF format, see "NetCDF Specifications"

  • In the GUI tkdegrib:
    Click on the "GIS" tab, and browse for your file. Double click on it in the top half, and it should fill out the inventory part in the bottom half. Select the message that you want in the bottom half, and choose the output name (or press "Recommend" to have tkdegrib "recommend" one. Next choose "NetCDF" for "File Type". Then press the "Generate NetCDF .nc file" button at the bottom.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -msg 1 -NetCDF 1
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, and extract the requested messages. The "-NetCDF 1" option tells it to create a NetCDF file using degrib's conventions version 1. This attempts to use the "CF" convention. See the "degrib Man Page" (particularly the "CONVERT OPTIONS" section) for more details.

Convert a sub grid of certain types of GRIB2 files to GRIB2:

You can use degrib to get a sub grid of certain types of GRIB2 files. It only works for GRIB2 files that were packed using simple pack, complex pack, or complex pack with second order differencing (which NDFD uses, but NCEP does not), as those are the only forms that it knows how to pack. Note you can't use this to convert a message from GRIB1 to GRIB2, because degrib currently doesn't have a mapping of the GRIB1 meta data to GRIB2 meta data.

  • In the GUI tkdegrib:
    Currently there is no way to do this from the GUI.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -C -msg 1 -Grib2 -lwlf 30,-105 -uprt 40,-85
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, and extract the requested messages. The "-Grib2" option tells it to create a GRIB2 message. The "-lwlf 30,-105" tells it to use a subgrid with the lower left corner at lon=-105E, lat=30N. The "-uprt 40,-85" tells it to use a subgrid with the upper right corner at lon=-85E, lat=40N. See the "degrib Man Page" (particularly the "CONVERT OPTIONS" section) for more details.

Create a flat file database:

You can use degrib to create a flat file database which is made up of either a bunch of .flt files and an index file (.ind), or a single cube file (.dat) and an index file (.ind). This is useful for CGI access to the NDFD, since probing the uncompressed form of the data is faster than probing the GRIB2 files.

  • In the GUI tkdegrib:
    Currently there is no way to do this from the GUI.

  • From the command line:
    /degrib/bin/degrib "GRIB file" -Data -Index simple.ind
    /degrib/bin/degrib "GRIB file" -Data -Index simple.ind -out simple.dat
    where "GRIB file" is replaced with the name of your GRIB file.

    This should read the file, create a new file "simple.dat", append each converted grid to simple.dat, and create an index file simple.ind. The "-Data" option tells it to create the "database", and the "-Index simple.ind" tells it what to name the index file. The "-out simple.dat" tells it what to name the ".dat" file. By default it uses the index filename replacing the extension with ".dat". See the "degrib Man Page" (particularly the "DATABASE CREATE OPTIONS" section) for more details.

Probe a flat file database:

You can use degrib to probe a flat file database which is made up of either a bunch of .flt files and an index file (.ind), or a single cube file (.dat) and an index file (.ind). This is useful for CGI access to the NDFD, since probing the uncompressed form of the data is faster than probing the GRIB2 files.

  • In the GUI tkdegrib:
    Currently there is no way to do this from the GUI.

  • From the command line:
    /degrib/bin/degrib "index file" -DP -pnt 35,-76
    where "index file" is replaced with the name of the index file (.ind) that you created when using the "-Data" command.

    This should read the index file, figure out where 35,-76 is, open the (.dat) file or the set of (.flt) files, jump to and read that cell, and return the results on the command line. See the "degrib Man Page" (particularly the "DATABASE PROBE OPTIONS" section) for more details.

Calculate a StormTotal:

Calculate the total amounts from a specified range of time. This is intented for QPF, SnowAmount, or IceAccum data when you want the total precipitation for an event.

  • In the GUI tkedegrib:
    Currently there is no way to do this from the GUI.

  • /degrib/bin/degrib "GRIB file" -StormTotal -startTime "timestamp" -endTime "timestamp" -Grib2
    where "GRIB file" is the name of your GRIB file, and the two timestamps are replaced with times such as 2012-05-31T12:00:00 and 2012-06-01T00:00:00.

    This should create a new GRIB2 file by combining any GRIB messages in the original file between the specified points in time. Keep in mind that the inventory option shows the end of the time frame and these times are exclusive. For example if the data is for 6 hour chunks, then to include 2012-06-01T18:00:00 in the output, you'd need a start time of 2012-06-01T12:00:00.

Output NDFD data in XML:

You can use degrib to convert National Digital Forecast Database (NDFD) GRIB files into Extensible Markup Language (XML). Currently, the only dialect of XML supported is Digital Weather Markup Language (DWML). You can find more information about DWML by reviewing its specification document (pdf) or its schema. You can tell degrib to format DWML encoded NDFD data and send it to standard output using the command line template below.

  • In the GUI tkdegrib:
    • Currently there is no way to do this from the GUI.
  • From the command line:

    Quick Start
    • executable   GRIB_files_dir   probeType   -pnt latidute,longitude   -XML optNum   -ndfdConven optNum   -ndfdVars elemList   -Icon optNum   -startTime timeString   -endTime timeString   -geoData GeoDataDir

      Example: /degrib/bin/degrib . -P -pnt 38.98,-77.07 -XML 1 -ndfdConven 2 -Icon 1 /degrib/data/geodata

      For more information on each command line option refer to the "degrib Man Page"
    Step-by-Step Instructions for Building the degrib Command to Generate DWML 
     

    We are going to use the command template shown in the "quick start" section to guide the building of a command to generate DWML encoded NDFD data. Follow each step in the table below to build the degrib command.

    Steps to Build the degrib CommandTemplate ComponentWhat your input might look like
    Locate the degrib executable and note its path. The executable name and path begin the command line input.executable/degrib/bin/degrib /degrib/bin/degrib_DP
    Locate the NDFD data. The directory containing the data is GRIB_files_dir in the command template.GRIB_files_dir/degrib/data
    Determine how you must probe the data. If you have files containing GRIB messages, then you will use the -P option. If, on the other hand, you have files called "data cubes" that contain float values that have been decoded from the original GRIB files, then you will use the -DP option.probeType-P
    Determine the latitude and longitude for each point that you want data for. You can enter more than one "-pnt latitude,longitude" option on the same command line if you want to probe more than one point at a time. You can also put all your points in file and reference it using the -pntFile [File] option (See "degrib Man Page" for more details).-pnt latidute,longitude-pnt 38.98,-77.07 -pnt 39.31,-77.83
    Determine which DWML product/format is appropriate for your data needs. The following samples may help you make the selection:

    -XML 1 - time-series product containing all NDFD & RTMA elements
    -XML 2 - glance product
    -XML 3 - 12 hourly summarization product
    -XML 4 - 24 hourly summarization product

    In addition to the numbered -XML convention, the following can be used:

    -XML time-series
    -XML glance
    -XML "12 hourly"
    -XML "24 hourly"    		-XML optNum-XML 1
    -XML "12 hourly"
    -XML glance
    Using the names of the data files as the ndfd element name, determine which ndfd elements you want in the output DWML. (See "degrib Man Page" the section on -ndfdConven for a list of element names). If you want all available ndfd elements in the DWML, then you don't have to include this option. Note, the -ndfdVars option is only applicable for the time-series (-XML 1) option.-ndfdVars elemList-ndfdVars wspd,temp,wx,sky,temp_r
    Determine if you want to see icon elements in the output DWML. If so set the -Icon option to 1. If you choose to format icon elements, be sure you have data files for probability of precipitation, sky cover, temperature, weather, and wind speed. These data files are needed by degrib to derive the icon elements. If you don't want icons in the DWML then you don't have to include this option. Icons by default are formatted for the "glance" product (-XML 2) and the "time-series" product (-XML 1) when there is no -ndfdVars argument.-Icon optNum-Icon 1
    Determine the time when degrib should start including NDFD in the output DWML. This is a UTC time. If you want the data for all available times, then you don't have to include this or the -endTime option.-startTime timeString-startTime 2006-12-25T23:00:00
    Determine the time when degrib should stop including NDFD in the output DWML. This is a UTC time. If you want the data for all available times, then you don't have to include this or the -startTime option.-endTime timeString-endTime 2007-01-01T08:00:00
    Locate your geographical data files and substitute the path for GeoDataDir-geoData GeoDataDir-geoData /degrib/data/geodata


    Putting it all together, we would have the following command for the "time-series" product:

    /degrib/bin/degrib /degrib/data -P -pnt 38.98,-77.07 -pnt 39.31,-77.83 -XML 1 -ndfdConven 1 -ndfdVars maxt,mint,pop12,qpf,wdir,wspd,temp,wx,sky -Icon 1 -startTime 2006-12-25T23:00:00 -endTime 2007-01-01T08:00:00 -geoData /degrib/data/geodata

    Example degrib Commands
     
    • Uses degrib to format a time-series of all NDFD elements for all times:

      /degrib/bin/degrib /degrib/data -P -pnt 38.98,-77.07 -XML 1 -ndfdConven 2 -geoData /degrib/data/geodata

    • Uses degrib to format a glance product using NDFD data for all times:

      /degrib/bin/degrib /degrib/data -P -pnt 38.98,-77.07 -XML 2 -ndfdConven 2 -geoData /degrib/data/geodata

    • Uses degrib to format a 12 hourly summarization product using NDFD data for all times:

      /degrib/bin/degrib /degrib/data -P -pnt 38.98,-77.07 -XML 12-hourly -ndfdConven 2 -geoData /degrib/data/geodata

    • Uses degrib to format a 24 hourly summarization product using NDFD data for all times:

      /degrib/bin/degrib /degrib/data -P -pnt 38.98,-77.07 -XML "24 hourly" -ndfdConven 2 -geoData /degrib/data/geodata

DeGRIB