CAVE Performance Preference

Purpose:

This jobsheet reviews how to change the CAVE Performance Preferences and how to monitor video card memory usage on the workstation using nvidia-smi. This is geared towards the ITO, but forecasters who monitor performance can benefit from learning a little about nvidia-smi. Any changes to the CAVE Performance Preferences on a live AWIPS should be coordinated with the ITO (10-15min).

Tasks:

  1. With the 2023 LX workstation replacements, the video cards are being increased to 5GB for WFOs and 16GB for National Centers.  To distribute the video card memory optimally among multiple CAVEs, the Video Card Texture Cache Size should be configured. The optimal setting is to take the video card memory (e.g. 5GB for at a WFO) and divide this by the number of CAVEs being used to display datasets that are GPU intensive like MRMS, radar mosaics, radar in the Volume Browser, and large frame counts of multi-panel radar/satellite data. Take that value and choose the next lowest option to ensure you do not eventually run out of video card memory. If you do use up all your video card memory, this will generate immediate unique memory alerts in AlertViz.
  2. Under the CAVE menu select Preferences.
  3. In the Preferences popup select Performance on the left hand side. Then select the Video Card Texture Cache Size pullout menu.
  4. For WFOs, the range of options are typically going to be relatively limited, so this ends up being pretty simple:
    • 1GB - supports up to 4 CAVEs
    • 2GB - supports up to 2 CAVEs
    • 4GB - supports 1 CAVE
      Note: Traditional guidance for the number of CAVEs per workstation running significant data visualization has been no more than 2 CAVEs. With the recent performance improvements and the 19" monitors getting replaced by a rotatable 27" monitor in the second half of 2023, forecasters might be tempted to use 3 CAVEs. For WFOs, sites using 3 CAVEs would potentially leave it at the 1GB default, but those using 2 CAVEs will set it to 2GB. Those only using 1 CAVE will set it to 4GB.
  5. Change the Video Card Texture Cache Size depending on the number of CAVEs you have loaded that are used for viewing data that is GPU intensive. Then click Apply and Close, and click on the OK menu.  Restart CAVE for the changes to take effect.
  6. You may wonder how much video card memory is being used on the workstation. To view how much video card memory you are using, you can run the nvidia-smi command at a terminal prompt:

    The Memory-Usage in the middle top column will tell you how much total video card memory the workstation is using (e.g. 312Mib/5036Mib. Instead of using MegaBytes (power of 10 based) they report in MebiBytes (power of 2 based). In the above example of a WFO configured LX workstation "5GB" graphics card, instead of that being represented as 5000 MB, the operating system actually sees this as 5036 MiB. 
    The Java process on the bottom is CAVE. With 1 CAVE only on the workstation and no data loaded, it is using 3 MebiBytes. There are other processes on this NoMachine session that are taking up memory that add up to 312Mib/5036Mib. 
  7. Load a 12 frame national radar reflectivity mosaic on conus scale from the Radar menu, and loop through the data. After the data has rendered each frame in the loop, the memory use should converge to a value. Run nvidia-smi. In this example at a regional HQ, the 12 frame radar mosaic takes up a whopping 3782Mib/5036Mib. 
  8. Clear the CAVE editor and run nvidia-smi again. Notice that much of the video card memory is freed up:
  9. Load an MRMS Rala reflectivity mosaic from the MRMS menu, and then run nvidia-smi. In this example the video card memory used increases from 345Mib to 846Mib for this 29 frame loop.
  10. Load some of your other favorite image-intensive data in CAVE like large frame counts of radar/satellite/model grid image multi-panels and note the video card memory usage. A single CAVE likely doesn't normally approach the limit of the video card memory, but adding multiple CAVEs will cause this to start adding up. If your office is considering increasing the number of CAVEs, you should do some performance testing to make sure there are no negative impacts. Using nvidia-smi can help inform you of what is using video card memory.
  11. Another type of memory usage that is involved with performance is the CAVE heap space. In 21.4.1, CAVE is allocated 8GB of memory at the WFOs and 31GB of memory at the National Centers. You can hover the mouse cursor over the memory usage in the bottom of CAVE to see more information on how much memory allocated for CAVE is being used.
  12. If your CAVE applications and data displays are using a lot of memory and not freeing it up, the memory widget on the bottom of CAVE will turn red, and you can run garbage collection by clicking on the trash can to free up any unused leftover memory.