Import Dam Info

Purpose:

This jobsheet demonstrates how information on dams is imported into Hazard Services, both from pre-existing WarnGen configurations and/or with manually maintained shapefiles

Tasks:

STEP 1: Checking for pre-configured dams in WarnGen

Most Focal Points will know whether their site already has pre-configured damBreak areas in WarnGen, but these steps can help confirm what you already have.

In the Localization Perspective, navigate to the Warngen sub-directory under the D2D folder.
Double click on damInfoBullet.xml to show the available versions
1. If you don't have this file, then either it is located somewhere else and you need to find it, or WarnGen was not configured for damBreaks
Double click on the SITE version of damInfoBullet.xml to open it for editing
1. For easier viewing, click on the "Source" tab at the bottom of your file
Check for the presence of un-commented dam entries, such as the following example:
```
<damInfoBullet bulletGroup="dam" bulletText="Big Rock Dam" bulletName="BigRockDam" parseString="BIG ROCK" showString="BIG ROCK DAM" coords="LAT...LON 3917 9405 3923 9387 3919 9385 3913 9394 3911 9405"/>
```
1. Fields have been highlighted which help identify the main dam entries and their pre-defined polygons.
2. Un-commented entries will not be embedded in  brackets, and will have multi-colored text in the Localization Perspective
Also check for the presence of "scenario" entries below each dam, such as the following example:
```
<damInfoBullet bulletGroup="scenario" bulletText="scenario - high fast" bulletName="BigRockhighfast" parseString="COMPLETE FAILURE OF BIG ROCK" showString="BIG ROCK DAM"/>
```
... where fields which help distinguish scenario entries from dam entries are highlighted.
1. These scenarios will also be imported by the parse script if present. If these don't exist, you can always specify them later in the Hazard Services metadata files, if desired.
If you have at least some dam entries with coordinates, then Hazard Services' parseWarngen script will be able to import them. If not, the parseWarngen script will not serve any purpose for you, but you can still begin creating your own shapefiles (see next item below) for use with the ingestShapeFiles script, and use Hazard Services to set up damBreaks for the first time.

STEP 2: Gathering Local Shape Files

Hazard Services facilitates loading local shapefiles into its maps database to use as pre-configured damBreak polygons (as well as for other impacts like burnScars).

(No action in this step) Designate a directory to maintain your shapefile collection in.
1. [FOR FOCAL POINT WORKSHOP] We'll use a directory on your forecaster desktop for now, like ~/Desktop/MyShapes/damBreak/
2. [FOR OPERATIONAL USE] A directory like /localapps/HazardServices/LLL/shapefiles/damBreak/ is recommended for several reasons:
  1. It isolates local shapefiles from any changes made to the /awips2 directory
  2. It allows shapefiles to be grouped by site LLL, which facilitates storing shapes for offices other than your own, such as for backup
  3. It separates shapefiles for damBreaks and burnScars into distinct folders, a necessary requirement for ingest later
If it doesn't already exist, create the directory you designated above as follows:
1. Open a terminal on your Linux, such as by right-clicking on the desktop and selection "Open Terminal"
2. Tailor the following command to match your desired directory [FOCAL POINT WORKSHOP: use ~/Desktop/MyShapes/damBreak] and execute it in the terminal:
```
mkdir -p /localapps/HazrdServices/LLL/shapefiles/damBreak # edit path to your desired 
```
[FOCAL POINT WORKSHOP] Generate at least one of your own shapefiles as follows:
1. Use the guidelines below and the jobsheet linked here to accomplish this
2. Use a unique but simple name (like 'myTestDam') for your test shape when prompted
3. As an output path, use the ~/Desktop/myShapes/damBreak directory you created previously
[FOR OPERATIONAL USE] Generally, create or consolidate shapefiles in the directory designated previously. As you do so, keep in mind that:
1. Shapefiles are allowed to contain one or more polygons
2. There should be a "name" attribute with each shapes, OR some other consistent attribute, to give the unique name of the corresponding dam
3. If no such attribute exists, the filename will be used by default, but this isn't as desirable
4. Every shapefile must have three varieties, one with each of the following extensions: .shp, .shx, .dbf
When finished creating or copying local shapefiles into your designated directory, move on to the next steps

STEP 3: Run ParseWarnGen

APPLIES ONLY if WarnGen has pre-configured dams (see Step 1 above). If not, skip this step.

Open a terminal window, for example by right-clicking on your desktop and selecting "Open in Terminal"
Save a session variable ("siteID") with the site ID whose WarnGen configurations you intend to import as follows (edit the fictitious site ID):
```
siteID=LLL # what site will you be importing for?
```
This variable will be used as an argument to the parse script later. Often this is simply your site, but parseWarnGen can be used to parse data for backup sites as well.
Verify the location of WarnGen's dam information files with the following steps:
1. Change directories to expected location of WarnGen's dam information files with the following command:
```
cd /awips2/edex/data/utility/common_static/site/${siteID}/warngen/
```
2. Ensure that both damInfoBullet.xml and damInfo.vm exist in this directory, using the following command to list the files out:
```
ls damInfo*{xml,vm} # If you see files, that's good
```
3. IF NOT, an alternate path will need to be manually specified later. Try to find any non-base files (if they exist) with a command like:
```
find /awips2/edex/ -name "damInfo.vm" -o -name "damInfoBullet.xml"
```
  ... keep in mind that you need to find files matching your chosen siteID. If a version of the files is found which is not in the common_static site directory, you can use that path.
4. Save a session variable ("damInfoPath") containing the actual path where damInfoBullet.xml is located. The command below is already set up for the customary location. If you did not find your files at the expected path in step 2, you should modify the path below first.
```
damInfoPath=/awips2/edex/data/utility/common_static/site/${siteID}/warngen/damInfoBullet.xml # Edit this path
```
  This variable will be used as an argument to the parse script next.
Finish setting up and run the parseWarnGen script with the following steps.
1. First, change directories to the location of Hazard Services' parseWarnGenTemplate script with the following command:
```
cd /awips2/edex/scripts/HazardServices/
```
2. If you have skipped assigning the $siteID and $damInfoPath variables above, you should do so now using the following commands (edit the fictitious siteID and path to match yours):
```
siteID=LLL # Site to use for import
damInfoPath=/awips2/edex/data/utility/common_Static/site/${siteID}/damInfoBullet.xml # WHERE is the corresponding site's damInfoBullet.xml?
```
3. Now, finally, run the parseWarnGenTemplate script by copy-pasting the following command into your terminal:
```
/awips2/python/bin/python parseWarngenTemplate.py -d ${siteID} ${damInfoPath}
```
  TECHNICAL NOTE: The above command is generalized to work for varying site and path scenarios. There is a simpler way to run parseWarnGen for damBreaks with only the -d argument as shown below...
```
/awips2/python/bin/python parseWarnGenTemplate.py -d
```
  ...HOWEVER this only works in a specific, standard case:
  1. You're running it for your own WFO (doesn't work for backup offices)
  2. Your damInfoBullet files are stored in the expected common_static/site/LLL/warngen directory
4. This is a good opportunity to review any warning messages or notifications... they may indicate improperly configured dams that could need attention once ported to Hazard Services.
To verify that parseWarnGen produced results for your imported site...
1. List the contents the expected output directory for shapefiles which will have been configured for your dams (based on the coordinates in the xml files seen previously), using the following command:
```
ll /awips2/edex/data/utility/common_static/configured/${siteID}/shapefiles/hazardServices
```
2. Do you have several damInfo files (with suffixes .dbf, .shp, .shx)? These are the shapefiles generated from your pre-configured warnGen damInfoBullet file. If not, there's been a problem, and you should refer to Section 1.0.2.4 of https://docs.google.com/document/d/1alM-U-KZJjicvZGnt0lWQ0zagDkzfiVtPR8QJUPG1oE/edit# for troubleshooting.
3. Now attempt to open the metadata created for your dams preferably by launching CAVE and using the Localization Perspective, in which you'll navigate to the Utilities sub-directory under the Hazard Services folder, then double-clicking on DamMetaData.py to see versions, then finally double-click on the Configured version for your newly imported site to see its contents.
  OR, you may use the following command open the file with vi in the terminal:
```
vim /awips2/edex/data/utility/common_static/configured/${siteID}/HazardServices/python/textUtilities/DamMetaData.py
```
4. Do you see a multi-level Python dictionary, with "DamMetaData = {" on the top, and one or more dams in the second tier, each with sub-dictionaries containing their own information? These data were also imported from damInfoBullet.xml and damInfo.vm.
5. (No action for this step) A note on the MetaData: Your auto-generated MetaData file may require some work, in particular if you plan to ingest local shapes in the next step which have no pre-existing configurations and will need to have new MetaData entries generated. If you need to fix your imported metadata, you need to create a site override. Refer to jobsheet linked here on managing dam/burnscar metadata overrides to do so.
For any issues with executing the parseWarnGenTemplate script, refer to Section 1.0.2.4 of https://docs.google.com/document/d/1alM-U-KZJjicvZGnt0lWQ0zagDkzfiVtPR8QJUPG1oE/edit# for troubleshooting.
NOTE: parseWArnGenTemplate.py is is likely only to be used once or a limited number of times during the transition

STEP 4: Run Shapefile Import

Whether you have just run the parseWarnGen script (which produces shapefiles from your WarnGen configured dams) or you have local shapefiles you want to import, this next step is the crucial step which imports the geometries into your maps database. The ingestShapeFiels script is capable of importing shapefiles from both sources (those parsed from WarnGen and other local ones) simultaneously.

Open a terminal if one isn't already open, for example by right-clicking on your desktop and selecting "Open in Terminal"
Save a session variable ("siteID") with the site ID for which you're importing shapes (edit the fictitious site ID):
```
siteID=LLL # what site will you be importing for?
```
1. Your terminal may already have this variable from the parseWarnGen steps, but it's repeated here for good measure.
2. This variable will be used as an argument to the ingest script later. Often this is simply your site, but ingestShapeFiles can be used to parse data for backup sites as well.
If you have local shapefiles to import (manually created or otherwise separate from any dams parsed from WarnGen), follow these steps:
1. Make sure you have properly separated dam and burnscar shape files into separate directories. (ANY shapefile in the path specified to the upcoming script will be imported into the database table whose type flag (dam or burnscar) you provide at run-time.)
  1. If you followed the advice to use /localapps/HazardServices/LLL/shapefiles/damBreak, then you are in good shape
  2. [FOCAL POINT WORKSHOP] The ~/Desktop/myShapes/damBreak directory we asked you to use is acceptable here
2. Save a session variable ("shapesPath") containing the actual path where your damBreak shapefiles have been consolidated. The command below is already set up for the recommended localapps directory... if you decided to use a different location, you should modify the path below first. Run this code in your terminal.
  [FOCAL POINT WORKSHOP] assign shapesPath=~/Desktop/myShapes/damBreak
```
shapesPath=/localapps/HazardServices/${siteID}/shapefiles/damBreak # Edit this path
```
Now, change directories to the location of Hazard Services' ingestShapeFiles script with the following command:
```
cd /awips2/edex/scripts/HazardServices/
```
Picking the scenario below which matches your circumstance, execute the associated command in your terminal to run the ingest shape files command...
1. WITHOUT ADDITIONAL SHAPE FILES:
```
./ingestshapefiles.sh -d -w ${siteID}
```
2. WITH ADDITIONAL SHAPE FILES:
```
./ingestshapefiles.sh -d -s ${shapesPath} -w ${siteID}
```
3. TECHNICAL NOTE: There are many other arguments and ways of using the ingestShapeFiles.sh script (run ./ingestShapeFiles.sh by itself to see usage). The scenarios above are intended to cover the most frequent need.
Note any key messages produced by the ingestShapeFiles script, such as:
1. "Existing impact areas are being preserved, unless they have a redundant name attribute with new shapes being ingested": This message lets us know that the default behavior of the ingestShapeFiles script, when it's run successive times, is to retain leave previously imported shapes in the database, UNLESS a shape with the same name is being imported again, in which case that shape is updated. This also means that you can run ingestShapeFiles multiple times, pointing at different shapefile directories (or perhaps shapes with different attribute structures, which it's good to keep separate), and each run will accumulate new shapes in the daminundation maps table.
2. "Some impact areas in mapdata.daminundation do not have an entry in any instance of DamMetaData.py": This message points to a helpful attempt by the ingest script to identify where new or existing shapes are missing critical MetaData. Every time the ingest script is run, a bare-bones DamMetaData.py file is auto-generated file with basic entries for these missing dams, which again encompasses any newly ingested shapes OR shapes already in the database which do not have an entry in the DamMetaData file. This auto-generated file (whose location and name is given after the above message) is suitable to use as or incorporate into an override file for DamMetaData.py, to provide at least minimal functionality for those shapes.
3. If you have any other issues, refer to Section 1.0.2.4 of https://docs.google.com/document/d/1alM-U-KZJjicvZGnt0lWQ0zagDkzfiVtPR8QJUPG1oE/edit# for troubleshooting
Before moving to the next step, carefully look for the "Some impact areas in mapdata.daminundation" message mentioned in the prior step...
1. If you see this mentioning a tmp/DamMetaData.py* file (with some numbers after it), that will be important!
2. This file, if created, is trying create a very basic metadata entry for any dams which do not yet have them
3. Feel free to open it for viewing, using the following gedit command as an example (make the numbers at the end of the filename match)
```
gedit /tmp/DamMetaData.py.00000 # make numbers at end match file given by message
```
Go on to the next step (checking what you've made), but keep in mind:
1. It will be necessary to manage damMetaData to correct values, add values (like scenarios) and more, especially if the shapes you ingested have never existed in the system before.

STEP 5: Check Results

The best way to check the result of this process is to try running the damBreak recommender and see if it has your new dams in it.

Launch Hazard Services
Run the Dam/Levee Break Flood Recommender
Look for your new/updated dam in the dropdown at the top of the dialog which appears
Click the Run button
Check the results
1. Does your polygon load and look the way you expected it to?
2. Do the other fields in the HID populate with fields like rivername, or scenarios, which are stored in the DamMetaData file?
If the polygon and/or HID reflect the updates you expected, congratulations! If not, you might need to re-do the polygon creation and ingest, and for non-polygon errors you may need to fix or add a DamMetaData override,

UPDATE OR ADD

Updating is fundamentally the same process as initial creation. A few notes:

You should not need to run parseWarnGen (step 3) again, unless you have updated your WarnGen files related to dam configurations. Most Focal Points will prefer to convert once to Hazard Services and stick to maintaining the latter.
As previously mentioned, the shapefile ingest script will accumulate shapes on successive runs, and will not delete existing shapes unless you manually tell it to. So you can feel comfortable importing additional shapefiles whenever you need to
If you need to UPDATE a shape, it's as simple as saving the shapefile with the same name attribute, and then ingesting again. When a new ingest has a matching name with an existing record, the existing record and its geometry are overwritten/updated

[FOCAL POINT WORKSHOP] Try repeating the shapefile creation and ingest steps, either for a new dam, or updating an existing dam (make sure you use the same "damName" variable when re-producing the shape), then confirm that you end up with additional dam choices or updated polygons, respectively.

DELETE A DAM BREAK IMPACT AREA

It may sometimes be necessary to remove a stored polygon (this may be more common for burnScars than damBreaks). This task demonstrates how to remove

Identify the dam whose polygon you wish to delete
1. [FOCAL POINT WORKSHOP] If you previously created a dummy shapefile (e.g. "myDam"), test this on that dam by assigning damName="your dam name"
ingestShapeFiles.sh -d -n $damName -w $siteID
NOTE: You may still need to delete the metadata for that removed dam

REFERENCES:

https://docs.google.com/document/d/1Yc8i0TfpP6z0TjrSm1aiJM5_UsMqoVWJuAiLzXuJAj0/edit

https://docs.google.com/document/d/1alM-U-KZJjicvZGnt0lWQ0zagDkzfiVtPR8QJUPG1oE/edit

Previous: Extension Area for Short-Fused	Focal Point Non-River Table of Contents	Next: Import Pre-existing Dam Areas From Warngen

Import Dam Info - Hazard Services

Focal Point Non-River Jobsheets