Purpose:

Tasks:

STEP 1 (of 4): Checking for pre-configured dams in WarnGen

Most Focal Points will know whether their site already has pre-configured damBreak areas in WarnGen, but these steps can help confirm what you already have. 

1. In the Localization Perspective, navigate to the Warngen sub-directory under the D2D folder.
2. Double click on damInfoBullet.xml to show the available versions
   1. If you don't have this file, then either it is located somewhere else and you need to find it, or WarnGen was not configured for damBreaks
3. Double click on the SITE version of damInfoBullet.xml to open it for editing
   1. For easier viewing, click on the "Source" tab at the bottom of your file
4. Check for the presence of un-commented dam entries, such as the following example:

   <damInfoBullet bulletGroup="dam" bulletText="Big Rock Dam" bulletName="BigRockDam" parseString="BIG ROCK" showString="BIG ROCK DAM" coords="LAT...LON 3917 9405 3923 9387 3919 9385 3913 9394 3911 9405"/>

   1. Fields have been highlighted which help identify the main dam entries and their pre-defined polygons.
   2. Un-commented entries will not be embedded in <!-- --> brackets, and will have multi-colored text in the Localization Perspective
5. Also check for the presence of "scenario" entries below each dam, such as the following example:

   <damInfoBullet bulletGroup="scenario" bulletText="scenario - high fast" bulletName="BigRockhighfast" parseString="COMPLETE FAILURE OF BIG ROCK" showString="BIG ROCK DAM"/>

   ... where fields which help distinguish scenario entries from dam entries are highlighted.
   1. These scenarios will also be imported by the parse script if present. If these don't exist, you can always specify them later in the Hazard Services metadata files, if desired.
6. If you have at least some dam entries with coordinates, then Hazard Services' parseWarngen script will be able to import them. If not, the parseWarngen script will not serve any purpose for you, but you can still begin creating your own shapefiles (see jobsheet on creating and importing your own shapes) for use with the ingestShapeFiles script, and use Hazard Services to set up damBreaks for the first time.

STEP 2 (of 4): Run ParseWarnGen

An automated script provided with Hazard Services does much of the work of converting dam info from the previous step into the files needed for the dam/levee flood recommender.  For each dam, coordinates are turned into shapefiles, and scenarios and other text are organized as entries in the DamMetaData.py file.

SOFTWARE NOTE: The scripts used in this and subsequent steps are delivered with AWIPS baseline and work for the majority of offices. However, updates and fixes may from time to time be provided within the Focal Point Guide (look for "postIOCUtilityScripts.tar") before they have been incorporated into new AWIPS installations. If you encounter problems with running these scripts, it is possible that using an updated version of the script may address those.

1. Open a terminal window as the awips user on the dx3 machine (if you are not the ITO or AWIPS focal point, you will want their assistance with this jobsheet)
   • Why dx3 and awips? dx3 hosts the baseline shell scripts (in /awips2/edex/scripts) which will be used to find and import warngen configurations, which is best done as the awips user for permissions reasons
2. Verify the location of WarnGen's dam information files with the following steps:
   1. The quickest way to cover all bases is to execute the "find" command below in your terminal:

      find /awips2/edex/ -name "damInfo.vm" -o -name "damInfoBullet.xml"
      

      .. keep in mind that you need to find files matching your own site (and not base files).
   2. Make note of the path where your site's damInfo.vm and damInfoBullet.xml are found. Usually the files will be located at /awips2/edex/data/utility/common_static/site/LLL/warngen. 
      In the unusual scenario where the files were not found in the standard location, their location will need to be provided to the next script as the ${damInfoPath} argument.
3. Now, execute the following command in your terminal to parse the data from your warngen files.

   /awips2/edex/scripts/HazardServices/parseWarngenTemplate.py -d

   1. This command uses a "-d" flag to indicate that we're parsing dam info
   2. TECHNICAL NOTE: The above command represents the simplest scenario for running parseWarnGen for dambreaks. However, there are other options for running the script (e.g. below), such as to specify offices other than your own (e.g. backup) or when the damInfo files were found in alternate locations.

      /awips2/edex/scripts/HazardServices/parseWarngenTemplate.py -d ${siteID} ${damInfoPath}

4. To verify that parseWarnGen produced results for your imported site...
   1. List the contents of the expected output directory using the command below, editing the LLL to match your site, to look for shapefiles which will have been configured for your dams (based on the coordinates in the xml files seen previously):

      ll /awips2/edex/data/utility/common_static/configured/LLL/shapefiles/hazardServices

   2. Do you have several damInfo files (with suffixes .dbf, .shp, .shx)? These are the shapefiles generated from your pre-configured warnGen damInfoBullet file. If not, there's been a problem, and you should refer to Section 1.0.2.4 of the Focal Point Guide for troubleshooting.
   3. Now attempt to open the metadata created for your dams preferably by launching CAVE and using the Localization Perspective, in which you'll navigate to the Utilities sub-directory under the Hazard Services folder, then double-clicking on DamMetaData.py to see versions, then finally double-click on the Configured version for your newly imported site to see its contents.
      NOTE: It may be necessary to close and re-launch CAVE to see this configured file appear
   4. Do you see a multi-level Python dictionary, with "DamMetaData = {" on the top, and one or more dams in the second tier, each with sub-dictionaries containing their own information? These data were also imported from damInfoBullet.xml and damInfo.vm.
   5. (No action for this step) A note on the MetaData: Your auto-generated MetaData file may require some work, and many fields such as pre-determined scenario text are possible to add here. If you need to fix your imported metadata, you need to create a site override. Refer to the jobsheet linked here on managing dam/burnscar metadata overrides to do so.
5. For any issues with executing the parseWarnGenTemplate script, refer to Section 1.0.2.4 of the Focal Point Guide for troubleshooting.
6. NOTE: parseWarnGenTemplate.py is is likely only to be used once or a limited number of times during the transition

STEP 3 (of 4): Run Shapefile Import

Whether you have just run the parseWarnGen script (which produces shapefiles from your WarnGen configured dams) or you have local shapefiles you want to import, this next step is the crucial step which imports the geometries into your maps database.  The ingestShapeFiles script is capable of importing shapefiles from both sources (those parsed from WarnGen and other local ones) simultaneously.

1. Open a terminal if one isn't already open, again as the awips user on the dx3 machine
2. Now, execute the following command in your terminal to ingest the shapefiles created by parseWarnGen into Hazard Services' map table 

   /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -d 

   1. This command uses a "-d" flag to indicate a new dambreak shape type. There is no argument needed for the path to the shapefiles because ingestshapefiles by default knows to look where parseWarnGenTemplate will have dumped shapefiles
   2. TECHNICAL NOTE: There are many other arguments and ways of using the ingestShapeFiles.sh script (run ./ingestShapeFiles.sh by itself to see usage). The scenarios above are intended to cover the most frequent need.
3. Note any help messages produced by the ingestShapeFiles script, such as:
   1. "Some impact areas in mapdata.hazardservicesarea do not have an entry in any instance of DamMetaData.py": This message points to a helpful attempt by the ingest script to identify where new or existing shapes are missing critical MetaData. Every time the ingest script is run, a bare-bones DamMetaData.py file is auto-generated file with basic entries for these missing dams, which again encompasses any newly ingested shapes OR shapes already in the database which do not have an entry in the DamMetaData file. This auto-generated file (whose location and name is given after the above message) is suitable to use as or incorporate into an override file for DamMetaData.py, to provide at least minimal functionality for those shapes.
   2. "Existing impact areas are being preserved, unless they have a redundant name attribute with new shapes being ingested": This message lets us know that the default behavior of the ingestShapeFiles script, when it's run successive times, is to leave previously imported shapes in the database, UNLESS a shape with the same name is being imported again, in which case that shape is updated. This also means that you can run ingestShapeFiles multiple times, pointing at different shapefile directories (or perhaps shapes with different attribute structures, which it's good to keep separate), and each run will accumulate new shapes in the daminundation maps table.
   3. If you have any other issues, refer to Section 1.0.2.4 of the Focal Point Guide for troubleshooting.
4. Before moving to the next step, carefully look for the "Some impact areas in mapdata.hazardservicesarea" message mentioned in the prior step...
   1. If you see this mentioning a tmp/DamMetaData.py* file (with some numbers after it), that will be important! 
   2. This file, if created, is trying create a very basic metadata entry for any dams which do not yet have them
   3. Feel free to open it for viewing using gedit or another text editor (Note your filename will be different)

      gedit /tmp/DamMetaData.py.00000 

5. [OPTIONAL] As long as you have a dx3 terminal open, a convenient verification of shapes that were imported is to use a slight variation of the ingestShapeFiles script to simply print "info" about the HazardServices maps table. 
   1. Execute the following command in your dx3 terminal to see a print-out of the various shapes within the "hazardservicesareas" maps table:

      /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -i

   2. Review the list of shapes that are printed by the script. The script displays the contents of the hazard services maps table grouped by shape type (e.g. dams, burnscars, etc.) so you can conveniently verify that the type you just imported reflects any new shapes or changes.
6. Go on to the next step (to check what you've imported within CAVE), but keep in mind:
   1. It will be necessary to manage damMetaData to correct values, add values (like scenarios) and more, especially if the shapes you ingested have never existed in the system before.

STEP 4 (of 4): Check Results

The best way to check the result of this process is to try running the damBreak recommender and see if it has your new dams in it.

1. Launch Hazard Services (either in D2D or Hydro perspective)
2. Run the Dam/Levee Flood Recommender (you may need to load a "Hydrology_ALL" or other setting to make this recommender visible)
3. Look for your new/updated dam in the dropdown at the top of the dialog which appears
4. Click the Run button
5. Check the results
   1. Does your polygon load and look the way you expected it to?
   2. Do you see pre-populated fields like "rivername" or scenarios. These will have been imported from your warngen templates if previously configured.
6. If the polygon and/or HID reflect the updates you expected, congratulations! If not, you might need to troubleshoot the shapefile ingest. For non-polygon errors, you may need to fix or add a DamMetaData override.

UPDATE OR ADD

Updating is fundamentally the same process as initial creation. A few notes:

• You should not need to run parseWarnGen (step 3) again, unless you have updated your WarnGen files related to dam configurations. Most Focal Points will prefer to convert once to Hazard Services and stick to maintaining the latter.
• As previously mentioned, the shapefile ingest script will accumulate shapes on successive runs, and will not delete existing shapes unless you manually tell it to. So you can feel comfortable importing additional shapefiles whenever you need to
• If you need to UPDATE a shape, it's as simple as saving the shapefile with the same name attribute, and then ingesting again. When a new ingest has a matching name with an existing record, the existing record and its geometry are overwritten/updated

DELETE A DAM BREAK IMPACT AREA

It may sometimes be necessary to remove a stored polygon (this may be more common for burnScars than damBreaks). This task demonstrates how to remove:

1. Identify the dam whose polygon you wish to delete. A convenient way to do this is to run ingestshapefiles (on dx3 as awips) with a simple -i flag (for printing information) as follows:

   /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -i

   ... once this script has printed the contents of the Hazard Services maps table (organized by shape type), you can find the exact name of the shape you'd like to delete (spaces, capitalization, etc. are all crucial to the subsequent step).
2. To  delete the shape you've identified, run the following command (where damName should be the exact name of your shape, surrounded by quotes to preserve any spaces, and siteID is an optional parameter for the CWA which the shape belongs in)

   /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -d -n $damName -w $siteID

3. NOTE: You may still need to delete the metadata for that removed dam