Standard Hydrological Exchange Format (SHEF)

This information does not come from MDL. The following originated from a website of the National Weather Service Office of Climate, Water, and Weather Services: Hydrologic Services Division. (http://www.nws.noaa.gov/oh/hrl/shef/shefdoc.htm )

Download SHEF Manual (PDF)

 

SHEF Software Instructions


Download Source Code

Download source code (.gz)

 

How to install the software

			DOWNLOADING SOFTWARE


          File Structure for the Shefpars Code Transfer (Linux)


A compressed tar file "ShefCode_tar.gz" is used to transfer the
shefpars code, test programs, and data.  Download this file and
uncompress it using the gunzip utility; then untar the file as follows:

      tar  -xvf  ShefCode_tar
      rm  ShefCode_tar

Seven directories and three text files are created as follows:

      README_shef_general .. readme text file with general information
      README_shef_install .. readme text file for installing the
                             software
      README_shef_updates... readme text file describing updates		
      shef_src ............. directory for all parsing subroutines
      shef_sheftest ........ directory for test mainline
      shef_data ............ directory for SHEFPARM file and example
                             input/output
      shef_lib ............. directory for parsing subroutines
                             archive library
      shef_scripts ......... directory for scripts to make makefiles
      shef_workspace ....... directory for running the test shefpars
                             program
      shef_shefit .......... directory for source to make program
                             "shefit"

All that is required to make a program using the shefpars decoder
is the source modules in directory "shef_src", and one data file
"SHEFPARM" in directory "shef_data".  Subroutine "shdriv" is the
driver for the parser.  The rest of the files are for support of
compiling and testing the parser.

A test program "shmain" and its called display routine "shpout" in
directory "shef_sheftest" can be used as an example of how to run the
shefpars decoder; or as an example of how to create a user developed
program calling routine "shdriv".  Its function is to open the input
message file and the "SHEFPARM" file, create and open the output
files, run the decoder, and display the results with a decoded text
file and a status/error file.

    Four scripts are included in directory "shef_scripts":

      make_lib_makefile ...... creates a makefile in directory
                               "shef_src" that will compile and make
                               an archived library for the shef
                               decoder.  The library is placed in
                               directory "shef_lib".

      make_sheftest_makefile . creates a makefile in directory
                               "shef_sheftest" that will compile the
                               "test" source modules and create an
                               executable called "shefpars" in
                               directory "shef_sheftest".

      make_shefit_makefile ... creates a makefile in directory
                               "shef_shefit" that will compile the
                               "shefit" source modules and create an
                               executable called "shefit" in
                               directory "shef_shefit".

      run_sheftest ........... makes a test run using the created
                               "sheftest" executable and test data
                               from directory "shef_data".  The
                               input data "shefin1", file "SHEFPARM",
                               and all the output files are placed
                               in directory "shef_workspace".

Note, the makefiles are created to work on our LINUX platform.  They
are about as generic as can be.  A "sed" function in the scripts that
make the makefiles attempts to create the needed environment.  Of
course, once the makefiles are created, they can be manually adjusted
to suit the user's needs before they are run.

The example test program "shefpars" requires one input line on the
standard input unit which contains the filename (128 char max) of
the input shef message to be decoded.  Also required is the shefpars
resource file "SHEFPARM" in the same directory where the test
program is run.  Three output files are created in the current
directory:

      shefout .... a binary output file for the edited input shef
                   message
      display .... a text file that displays the data in the binary
                   file (subroutine "shpout" creates this file)
      error ...... a status file with messages and errors (if any)

The following ksh script can be used independently to run "sheftest":

      Input=$2
      [[ -f ./SHEFPARM ]] || exit 1
      [[ -f $Input ]] || exit 1
      sheftest << EoF 2 > log
      $Input
      EoF

The following example output files are given with this program in
directory "shef_data":

      shefin1 .... a long example of shef input messages including
                   messages with errors
      SHEFdis .... the output from the "shpout" subroutine that
                   displays in text mode what the resulting "shefout"
                   file would contain
      SHEFerr .... the shef error messages from the given input file
                   (including a list of all input)

They can be compared with the output from the test run; but note that
some of the output dates are determined by when the program is run
since defaults to the current month or year are involved.

About program "shefit"; it is made using source code in directories
"./shef_shefit" and "./shef_src".  It is a filter version of the
shefpars program (output is ascii text).  For instructions on its
use, run the command "shefit -help" after making the program.


          Summary of Commands to Download and Run Shefpars


 1) Create a directory to contain the shefpars code (need about
    2 MB):

                cd  < parent-dir >
                mkdir  shefpars
                cd  shefpars

 2) Download the compressed tar file for the shefpars code
    (i.e. after ftp):

                cp  ShefCode_tar.gz  .

 3) Uncompress and untar to create a file structure:

                gunzip  ShefCode_tar.gz
                tar  -xvf  ShefCode_tar
                rm  ShefCode_tar

 4) Source code for the shefpars driver is in directory "shef_src".
    Now run some scripts to make makefiles; then run the makefiles
    to create a program; and finally run a script to test the
    program:

    Special Note1:  The "FFLAGS" and "CFLAGS" variables in the
                    makefiles are created below will probably need
                    editing for the local compilers.

                cd  ./shef_scripts
                make_lib_makefile
                make_sheftest_makefile

                cd  ../shef_src
                  < may need to edit file "Makefile" for compiler
                    options >
                  < may need to edit "shcurd.c" for underscore >
                make  -f  Makefile
                cd  ../shef_sheftest
                  < may need to edit file "Makefile" for compiler
                    options >
                make  -f  Makefile

                cd  ../shef_scripts
                run_sheftest

    Special Note2:  Source module "shcurd.c" may not compile with
                    some "Fortran-call-c" compilers ... so it may
                    be necessary to put a underscore "_" after the
                    name "shcurd" in the module "shcurd.c and
                    recompile it.

                 may need to change line:
                       void shcurd(int *yr,int *mo,int *da
                 to:
                       void shcurd_(int *yr,int *mo,int *da)
                 in routine "shcurd.c" and rerun the makefile.

 5) The output can be compared to the expected output in directory
    "shef_data" but note that some dates may differ because missing
    dates will default to a current date (later that when the
    example output was made):

                cd  ../shef_workspace
                diff  display  ../shef_data/SHEFdis
                diff  error  ../shef_data/SHEFerr

 6) Source code for the shefpars filter program "shefit" is in
    directory "shef_shefit".  It also uses the shefpars driver code.
    Run the following to make "shefit":

    Special Note3:  Source module "get_apps_defaults.c" may need a
                    underscore "_" at the end of the routine name
                    similar to module "shcurd.c" in Special Note2
                    above for some compilers.  For one compiler, two
                    underscores were needed; example:
                       "int get_apps_defaults__(..."

                cd  ../shef_scripts
                make_shefit_makefile

                cd  ../shef_shefit
                  < may need to edit file "Makefile" for compiler
                    options >
                  < may need to edit "get_apps_defaults.c" for
                    underscore >
                make  -f  Makefile
                shefit -help

Date:   01 April 2008

			
			

How to use the software


  THE DECODING OF SHEF MESSAGES USING THE "SHEFPARS" PARSING ROUTINES


                           ABSTRACT


Standard Hydrologic Exchange Format (SHEF) was developed as a
prerequisite for automated hydrologic data exchange.  This paper
describes how to use the software to parse shef messages and output
them to the desired database system.


                         TABLE OF CONTENTS


    1.  Introduction
    2.  Parsing and Posting
    3.  The Output File, SHEFOUT
    4.  Using Shefpars Source Code
    5.  Error Handling
    6.  Source Module Information


                         APPENDIX


    A.  Error Messages
    B.  Downloading Software



INTRODUCTION

Standard Hydrologic Exchange Format (SHEF) (1) is the standard
format used by the National Weather Service for encoding
hydrologic data.  SHEF is sufficiently flexible to handle most
hydro-meteorological data and as a result, is used by many programs
within the National Weather Service (NWS) as well as elements of
other government and private agencies.

SHEF has been designed to allow for the automation of data
handling techniques at the same time as maintaining a format which
is visually readable.

NWS handbook "Standard Hydrologic Exchange Format" documents the
syntax for SHEF messages.  Documentation and source code are available
from the NWS Office of Hydrology web page:

        http://www.nws.noaa.gov/ohd/hrl/shef/shefcode.htm

The following describes how to use the parsing subroutines in a
program to decode SHEF messages; how to obtain the output data
values; and how to download the source code.

PARSING AND POSTING

The process of taking data in SHEF and putting it in a database
can be described as a two step process; parsing and posting.  The
parsing step takes SHEF text messages and reduces them to a simplified,
machine readable format.  If a SHEF message contains several data
values, each is parsed into an separate output record ready for the
posting process.  The posting process takes the reduced data
and interacts with the target database in such a way that the data is
integrated into the database.

This conceptualization of the process allows software to be
developed for the parsing step which is essentially standard and is
not database specific thus allowing for a high degree of portability.
The software for the posting process is database specific and is not
presented here.

The software used by the National Weather Service reads SHEF
messages from a text file; passes each single decoded data value
and its attributes into an output subroutine, "shout", which writes
the value to a binary file with a set format that can be easily read
by a posting program.  All decoded values are passed to this one
subroutine, thus the form of the output from the shef parsing
routines can be changed by reprogramming this one subroutine.

Other files used by the parsing routines are a text resource file,
"SHEFPARM", and optionally one or two text output files to display
errors and a copy of the input messages.

THE OUTPUT FILE, SHEFOUT

This file is the "pipe" for passing the reduced data from the
parsing step to the posting step.  It is a Fortran sequential
unformatted file with each record being a self sufficient and complete
description of an item of data and its attributes.  All times on the
file have been converted to Universal Standard Time (i.e. GMT) and
all data is in English units as prescribed in SHEF.

Each record of the shefout file is written with the following
statement in subroutine, "shout":

        WRITE(LUOUT,IOSTAT=IERR)
     $        ID,IYR,IMO,IDA,IHR,IMN,ISE,KYR,KMO,KDA,KHR,KMN,KSE,
     $        PARCOD(1:1),BLNK3,PARCOD(2:2),BLNK3,IDUR,
     $        PARCOD(4:4),BLNK3,PARCOD(5:5),BLNK3,PARCOD(6:6),BLNK3,
     $        CODP,VALU,IQUAL,BLNK3,
     $        IREV,JID,ITIME,PARCOD,BLNK4,BLNK4,
     $        QUO

where:

    LUOUT ..... is the Fortran output file unit number
    BLNK3 ..... is three blank characters as CHARACTER*3
    BLNK4 ..... is four blank characters as CHARACTER*4

and data variables are:

    VARIABLE     TYPE           DESCRIPTION

    ID           CHARACTER*8    Station id
    IYR          INTEGER        Year of observation date (4 digits)
    IMO          INTEGER        Month of observation date
    IDA          INTEGER        Day of observation date
    IHR          INTEGER        Hour of observation date
    IMN          INTEGER        Minute of observation date
    ISE          INTEGER        Second of observation date
    KYR          INTEGER        Year of creation date (4 digits)
    KMO          INTEGER        Month of creation date
    KDA          INTEGER        Day of creation date
    KHR          INTEGER        Hour of creation date
    KMN          INTEGER        Minute of creation date
    KSE          INTEGER        Second of creation date
    PARCOD(1:1)  CHARACTER*1    First character of physical element
                                code
    PARCOD(2:2)  CHARACTER*1    Second character of physical element
                                code
    IDUR         INTEGER        Encoded duration code
    PARCOD(4:4)  CHARACTER*1    Type code
    PARCOD(5:5)  CHARACTER*1    Source code
    PARCOD(6:6)  CHARACTER*1    Extremum code
    CODP         REAL           Probability code
    VALU         DOUBLE PREC'N  Data value
    IQUAL        CHARACTER*1    Data qualifier
    IREV         INTEGER        Revision code (0=not a rev,1=rev)
    JID          CHARACTER*8    Data source
    ITIME        INTEGER        Time series indicator
                                (0=no ts,1=first elem,2=othr elem)
    PARCOD(1:8)  CHARACTER*8    Full parameter code
    QUO          CHARACTER*80   Internal comment (quote for data value)

Each output record contains 208 bytes; 128 bytes for the normal data
value variables and 80 bytes for the output internal comment or quote,
QUO.  The size of QUO is set in subroutine "shdcod" and can be changed
in that routine without any side effects in any other code.
For further description of the output variables check with the NWS
handbook, "Standard Hydrologic Exchange Format".

USING SHEFPARS SOURCE CODE

The parsing of SHEF messages is done by calling subroutine "shdriv".
The only arguments passed to "shdriv" are five Fortran unit numbers for
the input and output files used by the parser.  Only the first three
unit numbers are required, the last two output files are optional (a
"-1" will cause their output to be skipped).  Programs using the SHEF
parser routines assume the tasks of opening the files using Fortran
open statements, passing the unit numbers to "shdriv", and eventually
closing the files upon returning from "shdriv".  Inside "shdriv",
subroutine "shout" is eventually called which writes the decoded
data in binary format the the SHEFOUT file.  Subroutine "shout"
could be changed to accommodate the user's desired database system.

Following is a simplified example of how to use the shefpars routines
in a Fortran program (a C program would need to call a Fortran
subroutine to open the files and obtain Fortran unit numbers):

      INTEGER    LUINP,LUOUT,LUPAR,LUERR,LUDIS

      OPEN (LUINP,FILE='shef_input_message',STATUS='OLD')
      OPEN (LUOUT,FILE='shefout',FORM='UNFORMATTED',STATUS='NEW')
      OPEN (LUPAR,FILE='SHEFPARM',STATUS='OLD')

      OPEN (LUERR,FILE='shef_errors',STATUS='NEW')
      LUCPY = LUERR

        CALL SHDRIV(LUINP,LUOUT,LUPAR,LUERR,LUCPY)

      CLOSE(LUINP)
      CLOSE(LUOUT)
      CLOSE(LUPAR)
      CLOSE(LUERR)

where the following variables are needed for "shdriv":

    LUINP ... unit number of the text file to be decoded (the SHEF
              message "input" file),
    LUOUT ... unit number of the output binary file containing decoded
              data in a fixed format (the "shefout" file),
    LUPAR ... unit number of a text resource file containing shefpars
              decode parameters defining allowable codes (the
              "SHEFPARM" file),
    LUERR ... unit number of a file to contain error messages that may
              occur during the decoding,
    LUCPY ... unit number of a file to contain a copy of the input
              lines that are passed to the decoder.

Note that unit numbers LUERR and LUCPY can be set to -1 if those
output files are not needed, though it is recommended that the error
file should be used.  Also, if the output error file and display file
are to be combined into one file, just set LUCPY = LUERR instead of
opening a separate display file (as used in this example).  It is the
responsibility of the calling program to check that the input files
exist and that all files are opened correctly.

The currently used subroutine, "shout", outputs the decoded data to
the binary file ready for posting, using unit number LUOUT.  If the
user wants to look at the output, a new program or a subroutine would
be needed to read the binary file and display the data.

An alternate method of looking at the decoded message would be to
change subroutine "shout" to output the decoded data as text.  In this
case unit number LUOUT could be set to the standard output number for
Fortran, or it could be used to open a text output file if the write
statement in "shout" is modified to accompany it.

Subroutines shdura, shexcd, shfact, shmaxe, shpabg, shprob, shqual,
shsend, shtscd, and shyear each read the parameter file "SHEFPARM"
once each time the shef decode program is run; thus repetitively
executing the mainline to decode short messages will cause many reads
from the SHEFPARM file.  However, if the "shdriv" subroutine is
called many times from a continuously running mainline, the reading
of this resource file can be kept to a minimum.

The only other subroutine that reads data is "shline" in module shline.
It reads a single shef message line.  The following declaration line
in shline defines the length of an acceptable shef message line:

      CHARACTER*1004   LINE

The dimension is 4 more than the number of characters allowed in a
single message line.  Thus "CHARACTER*124" for example would be used
for a maximum message line length of 120 characters.

There is one "C" routine that calls C library routines "time" and
"localtime".  It is named "shcurd" and is called in routine "shyear"
that gets the current date.


ERROR HANDLING

All errors or warnings encountered during parsing are sent to an error
file through subroutine "sherr".  Most errors stop parsing the current
message while warnings are likely to output something.  All error and
warning messages are contained in subroutine "sherrm" and are output
to the error file through that routine.  Two other subroutines send
text output to the error file;  1) "sherrs" can be used to output a
summary of the number of errors and warnings;  2) "shvern" can be
used to output the version number of the parsing routines.  A copy of
each shef input line can be output through subroutine "shline" to a
separate file or the error file.

There are no provisions for handling signal interruptions.

SOURCE MODULE INFORMATION

All but one routine, "shcurd", is written in standard Fortran 77.
The C routine is in standard ANSI C.  All the routines come from RCS
files and have RCS keyword statements in them for tracking purposes
only, and could give warnings alluding to unused variables or
statements that cannot be reached.  They have never given any runtime
problems.

ERROR MESSAGES

This is a list of error and warning messages that comes directly
from subroutine "sherrm".

  001
  002   Two digits are required in date or time group
  003   An expected parameter code is missing
  004   File read error while accessing data file
  005   No dot in column 1 when looking for new message
  006   Dot found but not in column 1 of new message
  007   Unknown message type, looking for .A, .B, or .E
  008   Bad char in message type format (or missing blank delimiter)
  009   Last message format was different from this continuation messg
  010   Last message was NOT a revision unlike this continuation messg
  011   Last message had an error so cannot continue
  012   No positional data or no blank before it
  013   Bad character in station id
  014   Station id has more than 8 characters
  015   Bad number in positional data date group
  016   Incorrect number in date group
  017   Incorrect number in time group
  018   Missing blank char in positional data
  019   Bad creation date
  020   Bad date code letter after the character "D"
  021   Unknown data qualifier, data value is lost
  022   Unknown data units code (need S or E)
  023   Unknown duration code
  024   Bad 2-digit number following duration code
  025   Unknown time interval code (need Y,M,D,H,N,S,E)
  026   Bad 2-digit number following time interval code
  027   Bad character after "DR" (relative date code)
  028   Bad 1- or 2-digit number in relative date code
  029   Bad character in parameter code
  030   Bad parameter code calls for send code
  031   Trace for code other than PP, PC, PY, SD, SF, SW
  032   Variable duration not defined
  033   Bad character where delimiter is expected
  034   Non-existent value for given type and source parameter code
  035   ZULU, DR, or DI has send code QY, PY, or HY
  036   Forecast data given without creation date
  037   No value given after parameter code and before slash or eol
  038   Explicit date for codes DRE or DIE is not the end-of-month
  039   Year not in good range (1753-2199)
  040   Exceeded limit of data items
  041   Too many data items for given .B format
  042   Not enough data items for given .B format
  043   Cannot adjust forecast date to Zulu time
  044   Time between 0201 & 0259 on day changing from stnd to daylight
  045   No time increment specified (use DI code)
  046   No ".END" message for previous ".B" format
  047   ID requires 3 to 8 characters
  048   For DST, check Apr/Mar or Oct/Nov for 1976 thru 2040 only
  049   Bad character in the message
  050   Missing parameter code
  051   Bad value chars (or missing delimiter), data may be lost
  052   Bad character in data field
  053   "?" not accepted for missing, use "M" or "+"
  054   Parameter code is too long or too short
  055   Missing delimiter between data type fields
  056   Missing delimiter after data type field
  057   Should use "/" after date, time, or other D-code; before data
  058   Parm codes PP and PC require decimal value
  059   Abort, cannot read "shefparm" file correctly
  060   Non-existent value for given duration parameter code
  061   Non-existent value for given extremum parameter code
  062   Non-existent value for given conversion factor parameter code
  063   Non-existent value for given probability parameter code
  064   Parameter code too short or field misinterpreted as param-code
  065   Comma not allowed in data field, data value is lost
  066   Date check for yr-mo-da shows bad date
  067   No data on line identified with a message type format
  068   An unexpected ".END" message was encountered
  069    BUMMER!!!  Maximum number of errors reached, abort message
  070   Cannot output to binary shefpars file
  071   Cannot access "PE conversion factors" from the "shefparm" file
  072   Cannot access "send codes" from the "shefparm" file
  073   Cannot access "duration codes" from the "shefparm" file
  074   Cannot access "type/source codes" from the "shefparm" file
  075   Cannot access "extremum codes" from the "shefparm" file
  076   Cannot access "probability codes" from the "shefparm" file
  077   Cannot read "SHEFPARM" file!!!!!
  078   Bad character in data value, data value is lost
  079   Julian day should be written with 3 digits
  080   Too many digits in date group!
  081   Too many characters in quotes
  082   Data line found before completing .B format line(s)
  083   Missing slash delimiter or bad time zone code
  084   Too many chars in qualifier code, data value is lost
  085   Bad data qualifier, rest of format is lost
  086   Retained comment found without a data value, comment is lost
  087   Unexpected slash found after parameter code, before data value
  088   Cannot access "qualifier codes" from the "shefparm" file
  089
  090   Unknown error number given

Date:   14 July 2004

MOS