Purpose:

Tasks:

STEP 1 (of 3): Generate Shapefiles Using Baseline Tool

For focal points who do not have pre-defined polygons in WarnGen (or who just want to make a new shape) this process is a quick way of making new shapefiles. It uses the baseline tool in D2D to draw a shape, then calls a conversion script to export the baseline to a shapefile. The script will require information about which baseline to export, the filename/directory to save to, and a name for the new shape.

REQUIRED SCRIPT: baseline2shape.py (available on VLab here)

1. Download or locate a copy of the baseline2shape.py script so it can be used in the subsequent steps. Our instructions assume you save it to your Desktop, but it can be placed anywhere (just make a note of where it is so you can adjust the command accordingly when it's used)
2. In the D2D perspective, launch the baselines tool by clicking on its icon in the D2D toolbar, or selecting "Baselines" from the "Tools" menu
3. Zoom out to see both endpoints of the "A" baseline by using the middle-mouse scroll wheel
4. Find the "A" baseline to use for drawing
   1. You can use any baseline, but our commands later use baseline "A" when exporting coordinates, so you'll need to substitute your baseline at that time
5. Form a rudimentary polygon with your baseline as follows:
   1. Left-click and drag on the vertices to move them, making sure to keep the vertices inside the CWA
   2. Middle-click on a portion of the baseline to create additional vertices (you must have at least three vertices)
   3. Use zoom and other aides like maps to help better localize your polygon to an area of interest
   4. It is not necessary to close the shape, but keep in mind that the unconnected ends will be bridged when converted to a shapefile
6. Once satisfied with the shape of your baseline, open a terminal window on your LX, for example by right-clicking on your desktop and selecting "Open in Terminal"
7. Make sure a cwa-specific directory exists for exporting your dam shapefile to by running the command below in your terminal, editing the LLL to match your site

   mkdir -p /localapps/HazardServices/LLL/shapefiles/damBreak

   If you encounter permissions errors with this command, you may need to get help from your ITO or AWIPS Focal Point to either 1) have them create and allow group-write permissions on this directory, or 2) switch your terminal session to the awips user, so that you can create files in this directory in the next steps.
   CONVENTION NOTE: A directory like /localapps/HazardServices/LLL/shapefiles/damBreak/ is recommended for consolidating shapefiles several reasons.
   • It isolates local shapefiles from any changes made to the /awips2 directory
   • It allows shapefiles to be grouped by site LLL, which facilitates storing shapes for offices other than your own, such as for backup
   • It separates shapefiles for damBreaks and burnScars into distinct folders, a necessary requirement for ingest later
8. [DO NOT EXECUTE COMMAND YET] Prepare to run a conversion command (below) which turns your baseline into a shapefile. The command below accomplishes this after appropriate substitutions are made to the color-coded expressions

   /awips2/python/bin/python ~/Desktop/baseline2shape.py ${username} ${outputFilePath} ${baselineName} ${shapeName}

   1. ${username}: Substitute with your username on the LX (more precisely, the username you were logged in with when you created the baseline in D2D)
      • EXAMPLE: ejacobsen
      • If you don't provide this, the script will use whichever user your terminal is set to as a guess
   2. ${outputFilePath}: Substitute with the full path for the shapefile to create (including the name)
      • EXAMPLES: /localapps/HazardServices/LLL/shapefiles/damBreak/MyExampleDam  ...where LLL should be your CWA
      • Use a full path, ending in the desired filename (this is easily forgotten, so be careful to include this... the filename should not include .shp or any other extensions), to save the files in the place you want them
   3. ${baselineName}: Substitute with the name of the baseline you used
      • EXAMPLE: A
      • Only the letter is needed, no quotes or other surrounding characters
   4. ${shapeName}: Substitute with the name you want for your shape in the database
      • EXAMPLE: "My Example Dam"
      • Wrap in quotes, especially if you use spaces in the name
      • WHAT NAME TO USE? This will be the name added to your maps table as well as a default name in products, however you'll have a chance to specify an alternate in MetaData files later
   5. FULL EXAMPLE OF COMMAND:

      /awips2/python/bin/python ~/Desktop/baseline2shape.py ejacobsen /localapps/HazardServices/LLL/shapefiles/damBreak/MyExampleDam A "My Example Dam"
      

      • Again, make sure to include at the end of the path your shapefile name, e.g. "MyExampleDam"
9. After typing a customized version of the command above in your terminal to suit your shape, hit ENTER to execute the conversion command
   1. If successful, there should be NO message displayed
   2. If you get a "can't open file 'baseline2shape.py'" error, that means the baseline2shape.py script cannot be found at ~/Desktop/baseline2shape.py. Adjust the path "~/Desktop/baseline2shape.py" in the previous command to point to the correct location and try running again.  For example, if you switched to the awips user before running this command, you may not have access to your previous user's Desktop where you saved the baseline2shape.py file... in this case you should get help copying it (e.g. with scp) to the awips user's Desktop before trying the command again.
10. You're finished creating your shapefiles!
    1. Remember the output directory you used (e.g. "/localapps/HazardServices/...") ... that will be a crucial input when using ingestshapefile script to load your shapefile into AWIPS

STEP 2 (of 3): Run Shapefile Import

Whether you have just run the parseWarnGen script (which produces shapefiles from your WarnGen configured dams) or you have local shapefiles you want to import, this next step is the crucial step which imports the geometries into your maps database.  The ingestShapeFiles script is capable of importing shapefiles from both sources (those parsed from WarnGen and other local ones) simultaneously.

SOFTWARE NOTE: The scripts used in this and subsequent steps are delivered with AWIPS baseline and work for the majority of offices. However, updates and fixes may from time to time be provided within the Focal Point Guide (look for "postIOCUtilityScripts.tar") before they have been incorporated into new AWIPS installations. If you encounter problems with running these scripts, it is possible that using an updated version of the script may address those.

1. Open a terminal window as the awips user on the dx3 machine (if you are not the ITO or AWIPS focal point, you will want their assistance with this jobsheet)
   • Why dx3 and awips? dx3 hosts the baseline shell scripts (in /awips2/edex/scripts) which will be used to import your shapefiles, and which is best done as the awips user for permissions reasons
2. Now, execute the following command in your terminal (editing the LLL to match your site) to ingest the shapefiles you created into Hazard Services' map table 

   /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -d -s /localapps/HazardServices/LLL/shapefiles/damBreak/

   1. This command uses a "-d" flag to indicate a new dambreak shape type, and an "-s" flag followed by the path to your shape (ANY shape at that path will also be imported).
   2. TECHNICAL NOTE: There are many other arguments and ways of using the ingestshapefiles.sh script (run ./ingestShapeFiles.sh by itself to see usage). The scenarios above are intended to cover the most frequent need.
3. Note any help messages produced by the ingestShapeFiles script, such as:
   1. "Some impact areas in mapdata.hazardservicesarea do not have an entry in any instance of DamMetaData.py": This message points to a helpful attempt by the ingest script to identify where new or existing shapes are missing critical MetaData. Every time the ingest script is run, a bare-bones DamMetaData.py file is auto-generated file with basic entries for these missing dams, which again encompasses any newly ingested shapes OR shapes already in the database which do not have an entry in the DamMetaData file. This auto-generated file (whose location and name is given after the above message) is suitable to use as or incorporate into an override file for DamMetaData.py, to provide at least minimal functionality for those shapes.
   2. "Existing impact areas are being preserved, unless they have a redundant name attribute with new shapes being ingested": This message lets us know that the default behavior of the ingestShapeFiles script, when it's run successive times, is to leave previously imported shapes in the database, UNLESS a shape with the same name is being imported again, in which case that shape is updated. This also means that you can run ingestShapeFiles multiple times, pointing at different shapefile directories (or perhaps shapes with different attribute structures, which it's good to keep separate), and each run will accumulate new shapes in the daminundation maps table.
   3. If you have any other issues, refer to Section 1.0.2.4 of the Focal Point Guide for troubleshooting
4. Before moving to the next step, carefully look for the "Some impact areas in mapdata.hazardservicesarea" message mentioned in the prior step...
   1. If you see this mentioning a tmp/DamMetaData.py* file (with some numbers after it), that will be important! 
   2. This file, if created, is trying create a very basic metadata entry for any dams which do not yet have them
   3. Feel free to open it for viewing by using gedit (note the numbers in your filename will be different):

      gedit /tmp/DamMetaData.py.00000 

5. [OPTIONAL] As long as you have a dx3 terminal open, a convenient verification of shapes that were imported is to use a slight variation of the ingestshapefiles script to simply print "info" about the HazardServices maps table. 
   1. Execute the following command in your dx3 terminal to see a print-out of the various shapes within the "hazardservicesareas" maps table:

      /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -i

   2. Review the list of shapes that are printed by the script. The script displays the contents of the hazard services maps table grouped by shape type (e.g. dams, burnscars, etc.) so you can conveniently verify that the type you just imported reflects any new shapes or changes.
6. Go on to the next step (checking what you've imported within CAVE), but keep in mind:
   1. It will be necessary to manage damMetaData to correct values, add values (like scenarios) and more, especially if the shapes you ingested have never existed in the system before.

STEP 3 (of 3): Check Results

The best way to check the result of this process is to try running the damBreak recommender and see if it has your new dams in it.

1. Launch Hazard Services (either in D2D or Hydro perspective)
2. Run the Dam/Levee Flood Recommender (you may need to load a "Hydrology_ALL" or other setting to make this recommender visible)
3. Look for your new/updated dam in the dropdown at the top of the dialog which appears
4. Click the Run button
5. Check the results
   1. Does your polygon load and look the way you expected it to?  Be aware that Hazard Services will have rounded your polygon lat/lons to a precision of 0.01, so this might cause your resulting shape to differ slightly from the way you initially drew it.
   2. With a new shapefile, you will not have any pre-populated fields like "rivername" or scenarios. These will require dam-specific configurations in the DamMetaData.py file
6. If the polygon and/or HID reflect the updates you expected, congratulations! If not, you might need to re-do the polygon creation and ingest, and for non-polygon errors you may need to fix or add a DamMetaData override.

DELETE A DAM BREAK IMPACT AREA

It may sometimes be necessary to remove a stored polygon (this may be more common for burnScars than damBreaks). This task demonstrates how to remove:

1. Identify the dam whose polygon you wish to delete. A convenient way to do this is to run ingestshapefiles (on dx3 as awips) with a simple -i flag (for printing information) as follows:

   /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -i

   ... once this script has printed the contents of the Hazard Services maps table (organized by shape type), you can find the exact name of the shape you'd like to delete (spaces, capitalization, etc. are all crucial to the subsequent step).
2. To  delete the shape you've identified, run the following command (where damName should be the exact name of your shape, surrounded by quotes to preserve any spaces, and siteID is an optional parameter for the CWA which the shape belongs in)

   /awips2/edex/scripts/HazardServices/ingestshapefiles.sh -d -n $damName -w $siteID

3. NOTE: You may still need to delete the metadata for that removed dam