AWIPS Bundle Converter Script READ ME FOR INITIAL CAPABILITY UPDATED: May 2019 ORIGINAL: April 2018 Eric Jacobsen NWS Warning Decision Training Division ============================================================= SUMMARY: This script facilitates the modification of WDTD's FY18 RAC workshop procedures to work for a field office.  It scans each bundle and prompts the user for a substitution only when a potential need for customization to your WFO is identified. This script is an alternative to manual editing (steps can be found here: https://vlab.ncep.noaa.gov/web/wdtd/racproc) and intended to significantly reduce the time needed for customization. FEATURES: --> Prompts: When a radar, satellite sector, warning product filter, or other resource which may be different for your WFO is found in a bundle (and hasn’t been specified in a command-line argument), a prompt is shown to allow correction. Most of the time, you just hit ENTER to accept the default --> Memorization: Substitutions are memorized the first time they're entered, and won't need to be re-entered when they appear again in other bundles GENERAL WORKFLOW: The general process of procedure modification includes: 1) downloading the procedure files to work with 2) using the modification script on a live AWIPS to make WFO-specific updates to the procedure 3) saving files to the AWIPS utility tree (e.g. by copying them in through the “Localization Perspective”) WHAT YOU'LL NEED: * New Bundle and Colormap files * The bundle modification scripts, and an LX machine to run them on (i.e. a machine with CAVE, due to required python libraries) * Permissions to save files to your AWIPS' localization * Two optional but highly-recommended dependencies (NSEA and LFI), detailed next ============================================================= Dependencies ============================================================= Two “local applications” from NOAA’s Software Collaboration Portal will be useful before converting and using the procedures. ** Be sure to coordinate with your ITO, ESA, or AWIPS focal point on any localapp installations 1) Near-Storm Environmental Awareness (NSEA) Cursor Readout USED: In procedures, for sampling key severe weather parameters IMPACT: If missing, bundles will lack enhanced parameter cursor sampling. Will also receive notices in alertviz each time a bundle with NSEA is loaded about grid resources missing TO GET: Visit https://vlab.ncep.noaa.gov/redmine/projects/nwsscp/wiki/NseaCursorReadout and follow simple download instructions on wiki to get files, then install steps in “Documentation and Install Instructions” 2) Local File Installer (LFI) USED: To streamline bundle conversion script, by retrieving information from your localization IMPACT: If missing, will see more manual prompts for info during bundle conversion. When LFI is missing, script will also try to use caveData, which may be less preferable than directly accessing your localization (as LFI does). TO GET: Visit https://vlab.ncep.noaa.gov/redmine/projects/nwsscp/wiki/LocalFileInstaller and follow the simple instructions in “Documentation” link for installation **** NOTES for ITO/ESA/FocalPoints Getting LocalApps from NOAA’s SCP **** Retrieving files from the SCP is a relatively simple process requiring a VLab account, and which uses a common utility called subversion (svn) to retrieve files from an online repository. When following the instructions for each localapp, keep these useful tips in mind: * svn authentication to VLab requires your LDAP credentials. If prompted for a password with a username other than your LDAPP, hit ENTER to fail the prompt until it asks you to enter a username, and then you can enter LDAP username, then password * Enter “no” to any questions about storing your password in plain-text INSTRUCTIONS FOR USING BUNDLE CONVERTER SCRIPT ============================================================= STEP 1 - Get the files ============================================================= All files and scripts can be retrieved from VLab at: https://vlab.ncep.noaa.gov/web/wdtd/racproc This page has:     * Three (3) procedure files (ending in .xml)     * Five (5) colormap files (ending in .cmap)     * Six (6) files in the python script group which support this automated modification script ============================================================= STEP 2 - Use the script to modify your bundle ============================================================= ~~~Helpful Tips Before You Begin~~~ 1) Enlarge the size of the terminal window as much as you can, to make the displays as helpful as possible 2) Think about the 4 radar substitutions you’ll use for the 4-panel, multi-radar bundles, and how you’d arrange them in a 4-panel display. EXAMPLE: For the OUN WFO, the KVNX and KFDR radars are northwest and southwest of the office, so it makes sense to position them in the top-left and lower-left panels as seen below, while the other two radars are central, with KTLX being slightly north of TOKC.     +----+----+     |KVNX|KTLX|     +----+----+     |KFDR|TOKC|     +----+----+ NOTE: Substitutions with TDWRs will result in some problems, since not all 88D products are available with TDWRs. ~~~USING THE SCRIPT~~~ Follow the instructions below to run and interact with the script. 1) RUN: In a terminal window, change to the directory of the downloaded scripts, and execute the script with arguments below, such as with the path to the procedure file you want to modify, your WFO, the autoGuess flag to skip redundant prompts, and the 4 radars you want to appear in CLOCKWISE orientation (in order: top-left, top-right, bottom-right, bottom-left) in your 4-panel displays. python Main.py --bundle /full_path/to/procedureFile.xml --wfo ${your_3letter_CWA_id} --autoGuess --radars ${radar_topL} ${radar_topR} ${radar_botR} ${radar_botL} EXAMPLE: python Main.py --bundle ~/Downloads/RAC/RAC-FY19-GIDSevere.xml --wfo OUN --autoGuess --radars kvnx ktlx tokc kfdr (Don’t worry, if you forget to provide arguments, the script will prompt you for needed values.) OTHER OPTIONS: - Running without “--autoGuess” is useful for understanding the modifications being made, as it gives you the chance to confirm each substitution - Running without “--radars" shows you an example of the 4-panel display before prompting you for which radars to use in which panels - Run without “--bundle" if you want to choose from the current directory FOR HELP: To see all options for the script, type: python Main.py --help 2) CHOOSE BUNDLES: When presented with a numbered list of the bundles in the procedure file, hit ENTER to modify ALL of the bundles in the procedure file. 3) RESPOND TO PROMPTS: Wait as the script searches for resources needing modification in the selected bundle(s), and respond to prompts to make substitutions if found.  A mockup of the loaded products layout is displayed for each bundle (except for maps). Refer to this to orient yourself to where each resource is located. —> [ACCEPTING MEMORIZED SUBSTITUTIONS] Prompts will pre-populate with substitutions you've previously made in this execution of the script, so hit ENTER to simply accept a value that is already shown, or EDIT it if you'd like to change it After each bundle has been scanned, verify its new name at the bundle renaming prompt. The script automatically guesses the appropriate name by checking against substitutions made in that bundle. Hit ENTER to accept, or make edits first and then hit ENTER. 4) EDIT NEW FILE NAME: At the end of editing all bundles, edit/confirm the new name you’d like to use for saving the entire procedure file. This will save the file and exit. OUTPUT: The new, modified file is saved in the same directory as the initially imported file, but with the new name you gave it. RE-RUN: If needed, re-run the script with the steps above for other procedure files (e.g. hydro) to edit those as well. ============================================================= STEP 3 - Save files to the AWIPS utility tree ============================================================= For RAC-Specific instructions on files that need to be copied into the Localization Perspective, see: https://vlab.ncep.noaa.gov/web/wdtd/racproc Your new bundles needs to be saved to AWIPS' localization before you can use them (as well as the new colormap files they use). To do so, you can import them into CAVE with the "Localization Perspective". Any import requires that you have enough permission to contribute to your site’s localization files (check with your AWIPS focal point if you have problems). ~~~IMPORT THROUGH CAVE~~~ 1) In CAVE, open the Localization perspective (from "CAVE" menu > "Perspective" sub-menu > "Localization") 2) In the "File Browser" panel of CAVE's Localization perspective, click on the triangles to expand D2D, then Procedures 3) Import a procedure file by either:     a) Dragging and dropping the file from your computer to the "Procedures" folder icon in the "File Browser" panel     b) Right click on the "Procedures" folder icon in the "File Browser" panel, select "Import File", then find the file on your computer, click on it, and hit "OK" 4) Scroll down in the Localization Perspective File Browser until you see the file you imported, and click on the triangle to its left to see it     --> You will see a version imported just now at the User level, with your username in parentheses (e.g. "User (ejacobsen)")     --> If you see a previously imported Site version of the same procedure, you will want to carefully consider whether or not to replace it in the following step 5) Right click on your user version of the file, and hover over "Move To", then select the Site option (e.g. "Site (OAX)") to promote it to the site level for everyone in the office to use     --> You may also skip this step, or select other levels, such as ‘workstation’ if you only want users on the current machine to access the file 6) Repeat steps 3-5 for any additional procedure you wish to import (e.g. Hydro, Severe, etc.). NOTE: The procedures won’t work fully until you copy in colormaps, as discussed next. 7) Next, to import colormaps, find and expand "CAVE" in the Localization perspective's File Browser by clicking on the triangle to its left, then find and expand "Color Maps" 8) Import a colormap file by either:     a) Dragging and dropping the file from your computer to the "Color Maps" folder icon in the "File Browser" panel     b) Right click on the "Color Maps" folder icon in the "File Browser" panel, select "Import File", then find the file on your computer, click on it, and hit "OK" 9) Find the file you just imported in the Localization Perspective File Browser, and click on the triangle to its left to see it     --> You will see a version imported just now at the User level, with your username in parentheses (e.g. "User (ejacobsen)")     --> If you see a previously imported Site version of the same colormap, you will want to carefully consider whether or not to replace it in the following step 10) If the colormap you imported was either "streamflow flash exp2.cmap" or "uflow10.cmap", then drag the user version you just imported and drop it over the "mrms" subdirectory of "Color Maps" in the Localization Perspective 11) Right click on your user version of the file, and hover over "Move To", then select the Site option (e.g. "Site (OAX)") to promote it to the site level for everyone in the office to use     --> You may also skip this step, or select other levels, such as ‘workstation’ if you only want users on the current machine to access the file 12) Repeat steps 8-11 for any additional procedures you wish to import (5 total: '1024 Velocity.cmap', '1024 Z.cmap', 'HailMeteorTrail.cmap', 'streamflow flash exp2.cmap', 'uflow10.cmap') 13) Finished! FINALLY: Open the procedure in D2D and load each bundle. Watch for missing data, and manually fix if needed. Some potential data issues to expect include: - Specialized radar products aren’t available for your substituted radar (e.g. DIAL radars or TDWRs). In this case, some panels might have nothing displayed.