Run SS3 on a server

Forums

RSS (Opens New Window)

Francisco Izquierdo, modified 3 Years ago.

Run SS3 on a server

Youngling Posts: 9 Join Date: 10/30/20 Recent Posts

Dear SS3 experts,

I need to compile SS3 source code to run models on a big server. I have seen that SS3 source code is available on github, so I wanted to ask you if the github version is the stable one and also if there are any other issues I should communicate to the server programmers.

Many thanks in advance,

Fran

Kathryn Doering, modified 3 Years ago.

RE: Run SS3 on a server

Youngling Posts: 57 Join Date: 2/19/19 Recent Posts

Hi Fran,

I think it would be best to use the last release, 3.30.16: https://github.com/nmfs-stock-synthesis/stock-synthesis/releases/tag/v3.30.16

The main branch on github is not always stable, as we are actively developing and pushing new code there. Note that ss 3.30.16 is compiled with ADMB version 12.2, so the same version should be used: http://www.admb-project.org/downloads/admb-12.2/ . We do have some notes on how to compile SS.

Also, there are Linux versions of SS compiled: https://vlab.noaa.gov/web/stock-synthesis/document-library/-/document_library/0LmuycloZeIt/view/11683431 , I'm not sure if those would work for the server or not.

Thanks,

Kathryn

Francisco Izquierdo, modified 3 Years ago.

RE: Run SS3 on a server

Youngling Posts: 9 Join Date: 10/30/20 Recent Posts

Hi Kathryn,

Thanks a lot for your suggestion. I think the server works on linux so that option will be perfect.

Regards,

Fran

El 25/03/2021 a las 15:56, Kathryn Doering escribió:

Hi Fran,

I think it would be best to use the last release, 3.30.16: https://github.com/nmfs-stock-synthesis/stock-synthesis/releases/tag/v3.30.16

The main branch on github is not always stable, as we are actively developing and pushing new code there. Note that ss 3.30.16 is compiled with ADMB version 12.2, so the same version should be used: http://www.admb-project.org/downloads/admb-12.2/ . We do have some notes on how to compile SS.

Also, there are Linux versions of SS compiled: https://vlab.noaa.gov/web/stock-synthesis/document-library/-/document_library/0LmuycloZeIt/view/11683431 , I'm not sure if those would work for the server or not.

Thanks,

Kathryn

--
Kathryn Doering Stock Synthesis Virtual Lab Forum https://vlab.noaa.gov/web/stock-synthesis/public-forums/-/message_boards/view_message/14223990 VLab.Notifications@noaa.gov

Santiago Cervino, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 11 Join Date: 3/26/21 Recent Posts

HI, we are again compiling now ss 3.30.18 in a big server (linux). Our test run seems to work fine with -nohess. However when we try to run it with the hessian it finished with the following lines:

"got Fmsy 3.33077 16030.6
finished benchmark, forecast, and sdreporting
6 5144 -log(L): 5828.08 Spbio: 170790 83421 cat 27.373
End of file encountered trying to read temporary gradient file
: Invalid argument
Read 0bytes from temp. grad. file
Error reading stack identifer for bbbb
Error closing file gradfil1.tmp"

Do you think this issue can be related with the compilation process? This run works fine in my computer

Ian Taylor, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 117 Join Date: 12/8/14 Recent Posts

Hi Santiago Cervino,

This is a new one. @uh-johnoel.ancheta, if you get notifications from VLab, maybe you can chime in on this error: "Error reading stack identifer for bbbb Error closing file gradfil1.tmp".

@kelli.johnson also reached out to Chris Grandin who has experience with SS3 on AWS.

ADMB creates the gradfil1.tmp file to store gradient information when it doesn't have enough memory allocated for those calculations. It's possible that you could avoid the error through the use of the -gbs command line setting as described at http://www.admb-project.org/documentation/faq/#memory. I can't remember, but ADMB may be creating an empty file even in cases where there is adequate memory, so I'm not sure this would be a solution anyway.

-Ian

Santiago Cervino, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 11 Join Date: 3/26/21 Recent Posts

Hi,

I've been trying differnt things. I can choose the amout of memory needed when I send a job to the computer. I've been incresing til the maximun allowed, that is 160GB, and now I've got a different error:

"...

finished benchmark, forecast, and sdreporting
6 5144 -log(L): 5828.08 Spbio: 170790 83421 cat 27.373
Error reading stack identifer for aaaa
Error closing file cmpdiff.tmp
Error closing file gradfil1.tmp"

"cmpdiff.tmp" has 190 MB

"gradfil1.tmp" has 0 MB

3.30.18 was compiled in the same way that 3.30.16, one year ago. However 3.30.16 works fine.

Santi

Richard Methot, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 219 Join Date: 11/24/14 Recent Posts

Sorry to learn that you are still having problems. This will be hard to diagnose because it seems specific to your configuration. Are you compiling 3.30.16 and .18 with same version of ADMB?

Santiago Cervino, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 11 Join Date: 3/26/21 Recent Posts

The compilation is made by the CESGA team. CESGA stands for Supercomputer Centre of Galicia (www.cesga.es). As far as I known they used the ADMB release 12.3 in both SS versions (16 and 18). The 3.30.16 worked fine but not the 3.30.18. We can run 3.30.18 in our personal computers but we cannot do it in the CESGA servers.

Kelli Faye Johnson, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 28 Join Date: 6/13/17 Recent Posts

Regarding compiling, we have been compiling SS3 using the following instructions on AWS: https://github.com/cgrandin/docker-projects/blob/master/ss/install_ss.sh using the most recent commit of ADMB or a commit associated with the last released version, i.e., 1973802. Hopefully it is helpful to compare your workflow to this one, unfortunately we have not been having any problems so I cannot be helpful with respect to the error messages you are getting. Have you tried running a different SS3 model, like the simple example to see if the problem is specific to your input files?

Santiago Cervino, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 11 Join Date: 3/26/21 Recent Posts

The examples we tried in CESGA computers were previously run in our personal computers with 3.30.18 without any problem. CESGA servers had installed 3.30.16 one year ago and it also run fine. I'm sending this instructions to the CESGA colleages. Le's see what happend now. Thanks!

Santiago Cervino, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 11 Join Date: 3/26/21 Recent Posts

Problem solved. My bad! I had a memory configuration in my personal account causing the error when temporal big files overpass some size. Now every seems to work. Many thanks for your prompt support!!

Ian Taylor, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 117 Join Date: 12/8/14 Recent Posts

Great to hear the issue got resolved. Please report any future issues on the supercomputer.

Also, if it would be easier to have the Stock Synthesis releases at https://github.com/nmfs-stock-synthesis/stock-synthesis/releases include an executable that works on the specific flavor of linux you are using, rather than have the CESGA team keep ADMB up to date and compile SS3, we can try to make that happen.

Santiago Cervino, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 11 Join Date: 3/26/21 Recent Posts

HI, I've passed your offer to CESGA and they realy appreciated it, but they say that CESGA policy is not usung binaries but compile on their onwn machines. Thanks again!

Richard Methot, modified 2 Years ago.

RE: Run SS3 on a server

Youngling Posts: 219 Join Date: 11/24/14 Recent Posts

That's a relief. Thanks for giving all of us a reminder that memory can become an issue for big model setups.

Run SS3 on a server - Public Forums - Stock Synthesis

Forums