SHEF Information - MDL
Standard Hydrological Exchange Format (SHEF)
This information does not come from MDL. The following originated from a website of the National Weather Service Office of Climate, Water, and Weather Services: Hydrologic Services Division. (http://www.nws.noaa.gov/oh/hrl/shef/shefdoc.htm )
SHEF Software Instructions
Download Source Code
How to install the software
DOWNLOADING SOFTWARE
File Structure for the Shefpars Code Transfer (Linux)
A compressed tar file "ShefCode_tar.gz" is used to transfer the
shefpars code, test programs, and data. Download this file and
uncompress it using the gunzip utility; then untar the file as follows:
tar -xvf ShefCode_tar
rm ShefCode_tar
Seven directories and three text files are created as follows:
README_shef_general .. readme text file with general information
README_shef_install .. readme text file for installing the
software
README_shef_updates... readme text file describing updates
shef_src ............. directory for all parsing subroutines
shef_sheftest ........ directory for test mainline
shef_data ............ directory for SHEFPARM file and example
input/output
shef_lib ............. directory for parsing subroutines
archive library
shef_scripts ......... directory for scripts to make makefiles
shef_workspace ....... directory for running the test shefpars
program
shef_shefit .......... directory for source to make program
"shefit"
All that is required to make a program using the shefpars decoder
is the source modules in directory "shef_src", and one data file
"SHEFPARM" in directory "shef_data". Subroutine "shdriv" is the
driver for the parser. The rest of the files are for support of
compiling and testing the parser.
A test program "shmain" and its called display routine "shpout" in
directory "shef_sheftest" can be used as an example of how to run the
shefpars decoder; or as an example of how to create a user developed
program calling routine "shdriv". Its function is to open the input
message file and the "SHEFPARM" file, create and open the output
files, run the decoder, and display the results with a decoded text
file and a status/error file.
Four scripts are included in directory "shef_scripts":
make_lib_makefile ...... creates a makefile in directory
"shef_src" that will compile and make
an archived library for the shef
decoder. The library is placed in
directory "shef_lib".
make_sheftest_makefile . creates a makefile in directory
"shef_sheftest" that will compile the
"test" source modules and create an
executable called "shefpars" in
directory "shef_sheftest".
make_shefit_makefile ... creates a makefile in directory
"shef_shefit" that will compile the
"shefit" source modules and create an
executable called "shefit" in
directory "shef_shefit".
run_sheftest ........... makes a test run using the created
"sheftest" executable and test data
from directory "shef_data". The
input data "shefin1", file "SHEFPARM",
and all the output files are placed
in directory "shef_workspace".
Note, the makefiles are created to work on our LINUX platform. They
are about as generic as can be. A "sed" function in the scripts that
make the makefiles attempts to create the needed environment. Of
course, once the makefiles are created, they can be manually adjusted
to suit the user's needs before they are run.
The example test program "shefpars" requires one input line on the
standard input unit which contains the filename (128 char max) of
the input shef message to be decoded. Also required is the shefpars
resource file "SHEFPARM" in the same directory where the test
program is run. Three output files are created in the current
directory:
shefout .... a binary output file for the edited input shef
message
display .... a text file that displays the data in the binary
file (subroutine "shpout" creates this file)
error ...... a status file with messages and errors (if any)
The following ksh script can be used independently to run "sheftest":
Input=$2
[[ -f ./SHEFPARM ]] || exit 1
[[ -f $Input ]] || exit 1
sheftest << EoF 2 > log
$Input
EoF
The following example output files are given with this program in
directory "shef_data":
shefin1 .... a long example of shef input messages including
messages with errors
SHEFdis .... the output from the "shpout" subroutine that
displays in text mode what the resulting "shefout"
file would contain
SHEFerr .... the shef error messages from the given input file
(including a list of all input)
They can be compared with the output from the test run; but note that
some of the output dates are determined by when the program is run
since defaults to the current month or year are involved.
About program "shefit"; it is made using source code in directories
"./shef_shefit" and "./shef_src". It is a filter version of the
shefpars program (output is ascii text). For instructions on its
use, run the command "shefit -help" after making the program.
Summary of Commands to Download and Run Shefpars
1) Create a directory to contain the shefpars code (need about
2 MB):
cd < parent-dir >
mkdir shefpars
cd shefpars
2) Download the compressed tar file for the shefpars code
(i.e. after ftp):
cp ShefCode_tar.gz .
3) Uncompress and untar to create a file structure:
gunzip ShefCode_tar.gz
tar -xvf ShefCode_tar
rm ShefCode_tar
4) Source code for the shefpars driver is in directory "shef_src".
Now run some scripts to make makefiles; then run the makefiles
to create a program; and finally run a script to test the
program:
Special Note1: The "FFLAGS" and "CFLAGS" variables in the
makefiles are created below will probably need
editing for the local compilers.
cd ./shef_scripts
make_lib_makefile
make_sheftest_makefile
cd ../shef_src
< may need to edit file "Makefile" for compiler
options >
< may need to edit "shcurd.c" for underscore >
make -f Makefile
cd ../shef_sheftest
< may need to edit file "Makefile" for compiler
options >
make -f Makefile
cd ../shef_scripts
run_sheftest
Special Note2: Source module "shcurd.c" may not compile with
some "Fortran-call-c" compilers ... so it may
be necessary to put a underscore "_" after the
name "shcurd" in the module "shcurd.c and
recompile it.
may need to change line:
void shcurd(int *yr,int *mo,int *da
to:
void shcurd_(int *yr,int *mo,int *da)
in routine "shcurd.c" and rerun the makefile.
5) The output can be compared to the expected output in directory
"shef_data" but note that some dates may differ because missing
dates will default to a current date (later that when the
example output was made):
cd ../shef_workspace
diff display ../shef_data/SHEFdis
diff error ../shef_data/SHEFerr
6) Source code for the shefpars filter program "shefit" is in
directory "shef_shefit". It also uses the shefpars driver code.
Run the following to make "shefit":
Special Note3: Source module "get_apps_defaults.c" may need a
underscore "_" at the end of the routine name
similar to module "shcurd.c" in Special Note2
above for some compilers. For one compiler, two
underscores were needed; example:
"int get_apps_defaults__(..."
cd ../shef_scripts
make_shefit_makefile
cd ../shef_shefit
< may need to edit file "Makefile" for compiler
options >
< may need to edit "get_apps_defaults.c" for
underscore >
make -f Makefile
shefit -help
Date: 01 April 2008
How to use the software
THE DECODING OF SHEF MESSAGES USING THE "SHEFPARS" PARSING ROUTINES
ABSTRACT
Standard Hydrologic Exchange Format (SHEF) was developed as a
prerequisite for automated hydrologic data exchange. This paper
describes how to use the software to parse shef messages and output
them to the desired database system.
TABLE OF CONTENTS
1. Introduction
2. Parsing and Posting
3. The Output File, SHEFOUT
4. Using Shefpars Source Code
5. Error Handling
6. Source Module Information
APPENDIX
A. Error Messages
B. Downloading Software
INTRODUCTION
Standard Hydrologic Exchange Format (SHEF) (1) is the standard
format used by the National Weather Service for encoding
hydrologic data. SHEF is sufficiently flexible to handle most
hydro-meteorological data and as a result, is used by many programs
within the National Weather Service (NWS) as well as elements of
other government and private agencies.
SHEF has been designed to allow for the automation of data
handling techniques at the same time as maintaining a format which
is visually readable.
NWS handbook "Standard Hydrologic Exchange Format" documents the
syntax for SHEF messages. Documentation and source code are available
from the NWS Office of Hydrology web page:
http://www.nws.noaa.gov/ohd/hrl/shef/shefcode.htm
The following describes how to use the parsing subroutines in a
program to decode SHEF messages; how to obtain the output data
values; and how to download the source code.
PARSING AND POSTING
The process of taking data in SHEF and putting it in a database
can be described as a two step process; parsing and posting. The
parsing step takes SHEF text messages and reduces them to a simplified,
machine readable format. If a SHEF message contains several data
values, each is parsed into an separate output record ready for the
posting process. The posting process takes the reduced data
and interacts with the target database in such a way that the data is
integrated into the database.
This conceptualization of the process allows software to be
developed for the parsing step which is essentially standard and is
not database specific thus allowing for a high degree of portability.
The software for the posting process is database specific and is not
presented here.
The software used by the National Weather Service reads SHEF
messages from a text file; passes each single decoded data value
and its attributes into an output subroutine, "shout", which writes
the value to a binary file with a set format that can be easily read
by a posting program. All decoded values are passed to this one
subroutine, thus the form of the output from the shef parsing
routines can be changed by reprogramming this one subroutine.
Other files used by the parsing routines are a text resource file,
"SHEFPARM", and optionally one or two text output files to display
errors and a copy of the input messages.
THE OUTPUT FILE, SHEFOUT
This file is the "pipe" for passing the reduced data from the
parsing step to the posting step. It is a Fortran sequential
unformatted file with each record being a self sufficient and complete
description of an item of data and its attributes. All times on the
file have been converted to Universal Standard Time (i.e. GMT) and
all data is in English units as prescribed in SHEF.
Each record of the shefout file is written with the following
statement in subroutine, "shout":
WRITE(LUOUT,IOSTAT=IERR)
$ ID,IYR,IMO,IDA,IHR,IMN,ISE,KYR,KMO,KDA,KHR,KMN,KSE,
$ PARCOD(1:1),BLNK3,PARCOD(2:2),BLNK3,IDUR,
$ PARCOD(4:4),BLNK3,PARCOD(5:5),BLNK3,PARCOD(6:6),BLNK3,
$ CODP,VALU,IQUAL,BLNK3,
$ IREV,JID,ITIME,PARCOD,BLNK4,BLNK4,
$ QUO
where:
LUOUT ..... is the Fortran output file unit number
BLNK3 ..... is three blank characters as CHARACTER*3
BLNK4 ..... is four blank characters as CHARACTER*4
and data variables are:
VARIABLE TYPE DESCRIPTION
ID CHARACTER*8 Station id
IYR INTEGER Year of observation date (4 digits)
IMO INTEGER Month of observation date
IDA INTEGER Day of observation date
IHR INTEGER Hour of observation date
IMN INTEGER Minute of observation date
ISE INTEGER Second of observation date
KYR INTEGER Year of creation date (4 digits)
KMO INTEGER Month of creation date
KDA INTEGER Day of creation date
KHR INTEGER Hour of creation date
KMN INTEGER Minute of creation date
KSE INTEGER Second of creation date
PARCOD(1:1) CHARACTER*1 First character of physical element
code
PARCOD(2:2) CHARACTER*1 Second character of physical element
code
IDUR INTEGER Encoded duration code
PARCOD(4:4) CHARACTER*1 Type code
PARCOD(5:5) CHARACTER*1 Source code
PARCOD(6:6) CHARACTER*1 Extremum code
CODP REAL Probability code
VALU DOUBLE PREC'N Data value
IQUAL CHARACTER*1 Data qualifier
IREV INTEGER Revision code (0=not a rev,1=rev)
JID CHARACTER*8 Data source
ITIME INTEGER Time series indicator
(0=no ts,1=first elem,2=othr elem)
PARCOD(1:8) CHARACTER*8 Full parameter code
QUO CHARACTER*80 Internal comment (quote for data value)
Each output record contains 208 bytes; 128 bytes for the normal data
value variables and 80 bytes for the output internal comment or quote,
QUO. The size of QUO is set in subroutine "shdcod" and can be changed
in that routine without any side effects in any other code.
For further description of the output variables check with the NWS
handbook, "Standard Hydrologic Exchange Format".
USING SHEFPARS SOURCE CODE
The parsing of SHEF messages is done by calling subroutine "shdriv".
The only arguments passed to "shdriv" are five Fortran unit numbers for
the input and output files used by the parser. Only the first three
unit numbers are required, the last two output files are optional (a
"-1" will cause their output to be skipped). Programs using the SHEF
parser routines assume the tasks of opening the files using Fortran
open statements, passing the unit numbers to "shdriv", and eventually
closing the files upon returning from "shdriv". Inside "shdriv",
subroutine "shout" is eventually called which writes the decoded
data in binary format the the SHEFOUT file. Subroutine "shout"
could be changed to accommodate the user's desired database system.
Following is a simplified example of how to use the shefpars routines
in a Fortran program (a C program would need to call a Fortran
subroutine to open the files and obtain Fortran unit numbers):
INTEGER LUINP,LUOUT,LUPAR,LUERR,LUDIS
OPEN (LUINP,FILE='shef_input_message',STATUS='OLD')
OPEN (LUOUT,FILE='shefout',FORM='UNFORMATTED',STATUS='NEW')
OPEN (LUPAR,FILE='SHEFPARM',STATUS='OLD')
OPEN (LUERR,FILE='shef_errors',STATUS='NEW')
LUCPY = LUERR
CALL SHDRIV(LUINP,LUOUT,LUPAR,LUERR,LUCPY)
CLOSE(LUINP)
CLOSE(LUOUT)
CLOSE(LUPAR)
CLOSE(LUERR)
where the following variables are needed for "shdriv":
LUINP ... unit number of the text file to be decoded (the SHEF
message "input" file),
LUOUT ... unit number of the output binary file containing decoded
data in a fixed format (the "shefout" file),
LUPAR ... unit number of a text resource file containing shefpars
decode parameters defining allowable codes (the
"SHEFPARM" file),
LUERR ... unit number of a file to contain error messages that may
occur during the decoding,
LUCPY ... unit number of a file to contain a copy of the input
lines that are passed to the decoder.
Note that unit numbers LUERR and LUCPY can be set to -1 if those
output files are not needed, though it is recommended that the error
file should be used. Also, if the output error file and display file
are to be combined into one file, just set LUCPY = LUERR instead of
opening a separate display file (as used in this example). It is the
responsibility of the calling program to check that the input files
exist and that all files are opened correctly.
The currently used subroutine, "shout", outputs the decoded data to
the binary file ready for posting, using unit number LUOUT. If the
user wants to look at the output, a new program or a subroutine would
be needed to read the binary file and display the data.
An alternate method of looking at the decoded message would be to
change subroutine "shout" to output the decoded data as text. In this
case unit number LUOUT could be set to the standard output number for
Fortran, or it could be used to open a text output file if the write
statement in "shout" is modified to accompany it.
Subroutines shdura, shexcd, shfact, shmaxe, shpabg, shprob, shqual,
shsend, shtscd, and shyear each read the parameter file "SHEFPARM"
once each time the shef decode program is run; thus repetitively
executing the mainline to decode short messages will cause many reads
from the SHEFPARM file. However, if the "shdriv" subroutine is
called many times from a continuously running mainline, the reading
of this resource file can be kept to a minimum.
The only other subroutine that reads data is "shline" in module shline.
It reads a single shef message line. The following declaration line
in shline defines the length of an acceptable shef message line:
CHARACTER*1004 LINE
The dimension is 4 more than the number of characters allowed in a
single message line. Thus "CHARACTER*124" for example would be used
for a maximum message line length of 120 characters.
There is one "C" routine that calls C library routines "time" and
"localtime". It is named "shcurd" and is called in routine "shyear"
that gets the current date.
ERROR HANDLING
All errors or warnings encountered during parsing are sent to an error
file through subroutine "sherr". Most errors stop parsing the current
message while warnings are likely to output something. All error and
warning messages are contained in subroutine "sherrm" and are output
to the error file through that routine. Two other subroutines send
text output to the error file; 1) "sherrs" can be used to output a
summary of the number of errors and warnings; 2) "shvern" can be
used to output the version number of the parsing routines. A copy of
each shef input line can be output through subroutine "shline" to a
separate file or the error file.
There are no provisions for handling signal interruptions.
SOURCE MODULE INFORMATION
All but one routine, "shcurd", is written in standard Fortran 77.
The C routine is in standard ANSI C. All the routines come from RCS
files and have RCS keyword statements in them for tracking purposes
only, and could give warnings alluding to unused variables or
statements that cannot be reached. They have never given any runtime
problems.
ERROR MESSAGES
This is a list of error and warning messages that comes directly
from subroutine "sherrm".
001
002 Two digits are required in date or time group
003 An expected parameter code is missing
004 File read error while accessing data file
005 No dot in column 1 when looking for new message
006 Dot found but not in column 1 of new message
007 Unknown message type, looking for .A, .B, or .E
008 Bad char in message type format (or missing blank delimiter)
009 Last message format was different from this continuation messg
010 Last message was NOT a revision unlike this continuation messg
011 Last message had an error so cannot continue
012 No positional data or no blank before it
013 Bad character in station id
014 Station id has more than 8 characters
015 Bad number in positional data date group
016 Incorrect number in date group
017 Incorrect number in time group
018 Missing blank char in positional data
019 Bad creation date
020 Bad date code letter after the character "D"
021 Unknown data qualifier, data value is lost
022 Unknown data units code (need S or E)
023 Unknown duration code
024 Bad 2-digit number following duration code
025 Unknown time interval code (need Y,M,D,H,N,S,E)
026 Bad 2-digit number following time interval code
027 Bad character after "DR" (relative date code)
028 Bad 1- or 2-digit number in relative date code
029 Bad character in parameter code
030 Bad parameter code calls for send code
031 Trace for code other than PP, PC, PY, SD, SF, SW
032 Variable duration not defined
033 Bad character where delimiter is expected
034 Non-existent value for given type and source parameter code
035 ZULU, DR, or DI has send code QY, PY, or HY
036 Forecast data given without creation date
037 No value given after parameter code and before slash or eol
038 Explicit date for codes DRE or DIE is not the end-of-month
039 Year not in good range (1753-2199)
040 Exceeded limit of data items
041 Too many data items for given .B format
042 Not enough data items for given .B format
043 Cannot adjust forecast date to Zulu time
044 Time between 0201 & 0259 on day changing from stnd to daylight
045 No time increment specified (use DI code)
046 No ".END" message for previous ".B" format
047 ID requires 3 to 8 characters
048 For DST, check Apr/Mar or Oct/Nov for 1976 thru 2040 only
049 Bad character in the message
050 Missing parameter code
051 Bad value chars (or missing delimiter), data may be lost
052 Bad character in data field
053 "?" not accepted for missing, use "M" or "+"
054 Parameter code is too long or too short
055 Missing delimiter between data type fields
056 Missing delimiter after data type field
057 Should use "/" after date, time, or other D-code; before data
058 Parm codes PP and PC require decimal value
059 Abort, cannot read "shefparm" file correctly
060 Non-existent value for given duration parameter code
061 Non-existent value for given extremum parameter code
062 Non-existent value for given conversion factor parameter code
063 Non-existent value for given probability parameter code
064 Parameter code too short or field misinterpreted as param-code
065 Comma not allowed in data field, data value is lost
066 Date check for yr-mo-da shows bad date
067 No data on line identified with a message type format
068 An unexpected ".END" message was encountered
069 BUMMER!!! Maximum number of errors reached, abort message
070 Cannot output to binary shefpars file
071 Cannot access "PE conversion factors" from the "shefparm" file
072 Cannot access "send codes" from the "shefparm" file
073 Cannot access "duration codes" from the "shefparm" file
074 Cannot access "type/source codes" from the "shefparm" file
075 Cannot access "extremum codes" from the "shefparm" file
076 Cannot access "probability codes" from the "shefparm" file
077 Cannot read "SHEFPARM" file!!!!!
078 Bad character in data value, data value is lost
079 Julian day should be written with 3 digits
080 Too many digits in date group!
081 Too many characters in quotes
082 Data line found before completing .B format line(s)
083 Missing slash delimiter or bad time zone code
084 Too many chars in qualifier code, data value is lost
085 Bad data qualifier, rest of format is lost
086 Retained comment found without a data value, comment is lost
087 Unexpected slash found after parameter code, before data value
088 Cannot access "qualifier codes" from the "shefparm" file
089
090 Unknown error number given
Date: 14 July 2004