Hi,
Sorry for the slow response – I’m not sure if this is still of interest. We routinely conduct retrospectives with our SS assessments now and typically use the SS_doRetro() function, as a shortcut,
and take advantage of all the plotting structures and outputs that are produced.
However, we routinely modify the SS input files produced by these functions and re-run them – to modify some of the defaults from SS_doRetro. There are a number of changes that could be made, including
modifying the bias adjustments, that Ian refers to below. We typically ignore that nuance. However, as I understand it, SS now automatically truncates the part of the bias adjustment that slopes off on the right hand side, either at the last year of recruitment
estimates or the last year of input data, so if the right hand bend in the bias adjustment happens early enough (the entry in the control file with comment
#_last_yr_fullbias_adj_in_MPD), this truncation may still result in some adjustment of the bias ramp as years of data are dropped out (perhaps?).
I’m sure Ian Taylor or others can correct any of these details that I may have misinterpreted.
Here is the standard warning indicating this “truncation”
bias adjustment ramp extends into forecast; biasadj set to 0.0 internally for forecast years
This raises two additional points where there could potentially be further adjustments to each of the retrospective runs. We address the first of these, but ignore the second.
(i)
The last year that recruitments are estimated (the entry in the control file with the comment
# last year of main recr_devs; forecast devs start in following year). I don’t think SS_doRetro adjusts this value in the control file. We go in
and adjust this manually, decreasing the value by 1 for each year of data that is left out of the model run. If you don’t do this and remove 5 years of data, the last recruitments being estimated can give weird recruitment spikes, as they are no longer being
constrained by any information on recruitment from the length and age data. We don’t think this is behaviour that you want to be examining in a retrospective analysis – so we adjust these last year of recruitment estimates values manually, re-run SS in each
sub-directory and then use the SS plotting (re-running this) that has been nicely set up by SS_doRetro
(ii)
Iterative reweighting. As age and length data are removed from the runs, in theory the standard iterative reweighting done to tune the input and output variances should be repeated. Typically,
we don’t repeat this procedure for any of the standard sensitivities that we do, such as varying h or M (this would take too long) nor do we do this for our retrospective analyses. It could be argued that this should be done, both for sensitivities and for
retrospectives. This is not something that could be easily automated in SS_doRetro, partly because there are different versions of iterative reweighting used in different assessments. This does beg the question of whether others will routinely iteratively
re-weight any standard sensitivities that are run for assessments? The extra time required to do this used to make it prohibitive (especially with manual iterative-weighting), but with automation of iterative re-weighting, it seems this is something that could
be done. My question is should it be done?
ciao
Jemery
From: Ian Taylor <VLab.Notifications@noaa.gov>
Sent: Thursday, 10 December 2020 5:05 AM
To: VLab.Notifications@noaa.gov
Subject: [Stock Synthesis - r4ss] SS_doRetro
Glad you got it working, Laurie. The SS_doRetro() function in r4ss just modified the starter file and re-runs SS, so just saves time but doesn't differ from going through those steps manually. All CPUE and comp data from after the chosen retro year are ignored
in those analyses, so removing them manually should have the identical effect. To confirm this, you could use a model for which the standard retro process works without error.
I should note that one subtlety about retrospective analyses in SS is the use of the bias adjustment, where the inputs provided for when the recruitment bias adjustment ramps down aren't automatically shifted as you remove data (and thus information about
the most recent recruitments). The line between the main and late/forecast recdevs is also unchanged. Thus, the retro model you get may not match the settings you would have chosen if you were setting up that model manually. Adding automatic adjustment is
on the wishlist but hasn't been prioritized so far.
--
Ian Taylor Stock Synthesis Virtual Lab Forum
https://vlab.noaa.gov/web/stock-synthesis/public-forums/-/message_boards/view_message/12846749
VLab.Notifications@noaa.gov
