List of Troubleshooting Items

• Generic Troubleshooting
  • Look through the various log files to find and diagnose problems/errors.
  • Once an issue has been identified and isolated begin diagnosing the situation.
  • Start with the most probable and common issue and progress to the least probable and uncommon issue.
  • Check the syntax when making changes to any GFE config or GFE server config file in the Localization Perspective.
  • Restart GFE.
  • Restart the GFE server process using the GFE Site Activation GUI or by manually running the scripts via the command line in a terminal to deactivate and then activate the site.
  • Start GFE via the command line with the noredirect switch in a terminal.
    • This will output any errors GFE encounters to the terminal.
  • Ask forecasters to save off AlertViz messages and send to you for evaluation and troubleshooting.
• Smart Init Troubleshooting
  • Look at all the methods and make sure they all have proper python syntax.
  • If they don't, then fix them so that they do and manually rerun the Smart Init using ifpInit.
• Formatter Troubleshooting
  • Text Products Troubleshooting
    • Run text products locally and view error messages in the Process Monitor.
    • Look at the console logs for help with diagnosing problems that occur when running the formatter launcher.
    • Look at the EDEX logs for issues that occur when configureTextProducts.py is run.
  • Formatter Fails
    • The above link goes to a Wiki created by Jonathan Lamb that explains how to perform troubleshooting when a formatter fails.
  • Getting Undesirable Results with the Baseline Formatter
    • Check the TextRules Reference documentation to see if there are thresholds and variables that you might set to produce desired results.
  • Getting Undesirable Results and You have Overridden some Thresholds and/or Variables in Your Overrides File
    • Run the baseline version of the product to see if you get the same results.
    • If so, use the strategy above.
    • If not, comment out the overrides in your Overrides file and run it as the baseline.
    • Now begin adding in the overrides one-by-one until the problem re-occurs.
  • Get the Error Message: WARNING: TOO MANY PASSES ON TREE or WARNING EMPTY WORDS FROM TREE
    • Isolate the edit area for which there is a problem. Run the product for that area only by altering your Combination file or the "defaultEditAreas" list.
    • Isolate the time period for which there is a problem. Override the issuance_list for the product and comment out all periods except the first for the issuance causing the problem. Run the product for this time period and see if you get the same error. If not begin uncommenting the periods one at a time until you get the error. For any components prior to the one causing the trouble override the product component definition and remove all the entries in the "analysisList" and "phraseList" variables.
    • Isolate the phrase method. Determine which phrase is causing the problem by overriding the product component definition for the time period causing the trouble. Comment out all but the first phrase method and run the product. If the problem does not occur, begin uncommenting each phrase one-by-one and running the product until you find which phrase is causing the problem. Then comment out all but that phrase method and run the product again to see that the problem occurs. In some rare cases, you may have to uncomment more than one phrase to get the problem to occur.
    • Check the offending phrase method code. Find the phrase "word" method in the TextRules Library (or your Overrides file if you have overridden or added methods) and ensure that all of its return statements return with a self.Done() call or a self.setWords() call. If not correct the code and run your product again. If that was not the problem, proceed to the next step.
    • Turn on the trace messages. Now you can turn on the trace messages in your Overrides file by setting the Definition["trace'] flag to run. Re-run the product and examine the end of the trace messages. You are looking for a method that is repeating without returning a status of 'done' as reported in the message or is continually returning None for the value of 'Words' in the trace message. If you find the offending method, you can begin to investigate why it is not completing.
    • Check your overrides. You will want to isolate which of your overrides introduces the problem.
  • Formatter Does Not Appear in Formatter Launcher
    • Make sure the formatter has the Display Name set to something other than None in the lowest level override, usually SITE-level, Definition or Text Product file. Also make sure there isn't an override in the lowest-level override, usually SITE-level, Overrides file (if it exists for the Text Product) that has the Display Name set to None.
  • Missing a Configured File (using the Hazard_AQA formatter as an example)
    • In the Localization Perspective under the EDEX Folder, under the AFOS2AWIPS2 Folder:
      • Make sure the line PHXAQAXXX AEUS75 KXXX is removed from the SITE version of afos2awips.blacklist.txt where XXX is your site id.
      • Make sure the line PHXAQAXXX AEUS75 KXXX is added to the SITE version of afos2awips.txt.
    • Check the afosMasterPIL.txt and textCategoryClass.txt files located in /awips2/edex/data/utility/common_static/configured/XXX/textdb/ where XXX is your site id.
      • Do this per the instructions in the AWIPS II System's Manager Manual (SMM) section 16.6.2.
      • Make sure PHXAQAXXX is in afosMasterPIL.txt.
      • Make sure AQA is in textCategoryClass.txt
    • Touch one of the formatter files in common_static/base/gfe/textproducts/templates/product/
      • Run in terminal touch /awips2/edex/data/utility/common_static/base/gfe/textproducts/templates/product/DGT.py .
    • Touch the localConfig.py file in the Localization Perspective under the GFE Server Folder under the Server Config Files Folder
      • Make a minor change, save it, and then remove the minor change .
    • Run in terminal /awips2/GFESuite/bin/configureTextProducts XXX as user awips on dv3, where XXX is your site id.
• ISC Troubleshooting
  • User Errors in localConfig.py
    • Ensure all necessary settings have been added.
    • Ensure there are no typos.
    • Make sure that the extraISCparms and REQUESTED_ISC_PARMS lists use plus equals (+=) and NOT equals (=). 
    • Validate your config using the instructions in the Server-side files section of the Configuration and Management section to check for syntax errors.
  • Use the IRT website within AWIPS to troubleshoot ISC problems.
• BOIVerify Troubleshooting
  • Missing Model Data
    • Check to see if the database is available in BOIVerifyConfig.py.
    • Check to see if the database is included in SAVE_MODELS in BOIVerifyConfig.py.
    • If the database is available and included in SAVE_MODELS, then there may be a problem with the cron or available storage space.
  • Bias Corrected Grids not Calculating
    • Check to see if model data is available in GFE.
    • Check to see if the entry to create the bias corrected database for the model is in localConfig.py.
    • Check to see if the model is listed in BiasCorrModels.txt
    • Check to see if there are enough days of archived grids to begin calculating bias corrected grids which is set in BOIVerifyConfig.py.
    • Check to see if there are observation grids stored for the element.
    • If the first five do not solve the problem, then there may be a problem with the cron or available storage space.
  • Data Missing in the Automated (BOIVerifySumAll.py/BOIVerifySumTemps.py) Summary
    • Check to see if the correct models are defined in the BOIVerifySumAll.py/BOIVerifySumTemps.py procedure.
    • Check to see if BOIVerifySumAll.py/BOIVerifySumTemps.py can create statistics for the desired element.
    • If the first two do not solve the problem, then there may be a problem with the cron or available storage space.
  • System Issues (cron, space)
    • Check to see if the AWIPS-2 apps package is running on pv2.
    • Check to see if the user password has expired.
    • Check to see if there is any storage space left on the drive.
• Service Backup Troubleshooting
  • Failed Site's Data on the Central Server.
    • If the failed site's data cannot be download, then there is a problem with the failed site's data on the central server.
    • If the failed site's grid database is incomplete, then there is a problem with the failed site's data on the central server.
    • If there are error messages or disabled functions in the failed site's GFE, then there is a problem with the failed site's data on the central server.
  • Backup site's Service Backup files.
    • If the site data cannot be imported, then there is a problem with the backup site's Service Backup files.
    • If Service Backup cannot be exited, then there is a problem with the backup site's Service Backup files.
    • If the system indicates you are backing up a site erroneously, then there is a problem with the backup site's Service Backup files.