Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods

J. J. Deroba, D. S. Butterworth, R. D. Methot, J. A. A. De Oliveira, C. Fernandez, Anders Nielsen, S. X. Cadrin, M. Dickey-Collas, C. M. Legault, J. Ianelli, J. L. Valero, C. L. Needle, J. M. O'Malley, Y.-J. Chang, G. G. Thompson, C. Canales, D. P. Swain, D. C. M. Miller, N. T. Hintzen, M. Bertignac & 15 others L. Ibaibarriaga, A. Silva, A. Murta, L. T. Kell, C. L. de Moor, A. M. Parma, C. M. Dichmont, V. R. Restrepo, Y. Ye, E. Jardim, P. D. Spencer, D. H. Hanselman, J. Blaylock, M. Mood, P.- J. F. Hulson

Research output: Contribution to journalJournal articleResearchpeer-review

Abstract

The World Conference on Stock Assessment Methods (July 2013) included a workshop on testing assessment methods through simulations. The exercise was made up of two steps applied to datasets from 14 representative fish stocks from around the world. Step 1 involved applying stock assessments to datasets with varying degrees of effort dedicated to optimizing fit. Step 2 was applied to a subset of the stocks and involved characteristics of given model fits being used to generate pseudo-data with error. These pseudo-data were then provided to assessment modellers and fits to the pseudo-data provided consistency checks within (self-tests) and among (cross-tests) assessment
models. Although trends in biomass were often similar across models, the scaling of absolute biomass was not consistent across models. Similar types of models tended to perform similarly (e.g. age based or production models). Self-testing and cross-testing of models are a useful diagnostic approach, and suggested that estimates in the most recent years of time-series were the least robust. Results from the simulation exercise provide a basis for guidance on future large-scale simulation experiments and demonstrate the need for strategic investments in the evaluation and development of stock assessment methods
Original languageEnglish
JournalICES Journal of Marine Science
Volume72
Issue number1
Pages (from-to)19-30
ISSN1054-3139
DOIs
Publication statusPublished - 2015

Cite this

Deroba, J. J. ; Butterworth, D. S. ; Methot, R. D. ; De Oliveira, J. A. A. ; Fernandez, C. ; Nielsen, Anders ; Cadrin, S. X. ; Dickey-Collas, M. ; Legault, C. M. ; Ianelli, J. ; Valero, J. L. ; Needle, C. L. ; O'Malley, J. M. ; Chang, Y.-J. ; Thompson, G. G. ; Canales, C. ; Swain, D. P. ; Miller, D. C. M. ; Hintzen, N. T. ; Bertignac, M. ; Ibaibarriaga, L. ; Silva, A. ; Murta, A. ; Kell, L. T. ; de Moor, C. L. ; Parma, A. M. ; Dichmont, C. M. ; Restrepo, V. R. ; Ye, Y. ; Jardim, E. ; Spencer, P. D. ; Hanselman, D. H. ; Blaylock, J. ; Mood, M. ; Hulson, P.- J. F. / Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods. In: ICES Journal of Marine Science. 2015 ; Vol. 72, No. 1. pp. 19-30.
@article{ee065f7796a84ad1ae3ea793582f15a7,
title = "Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods",
abstract = "The World Conference on Stock Assessment Methods (July 2013) included a workshop on testing assessment methods through simulations. The exercise was made up of two steps applied to datasets from 14 representative fish stocks from around the world. Step 1 involved applying stock assessments to datasets with varying degrees of effort dedicated to optimizing fit. Step 2 was applied to a subset of the stocks and involved characteristics of given model fits being used to generate pseudo-data with error. These pseudo-data were then provided to assessment modellers and fits to the pseudo-data provided consistency checks within (self-tests) and among (cross-tests) assessmentmodels. Although trends in biomass were often similar across models, the scaling of absolute biomass was not consistent across models. Similar types of models tended to perform similarly (e.g. age based or production models). Self-testing and cross-testing of models are a useful diagnostic approach, and suggested that estimates in the most recent years of time-series were the least robust. Results from the simulation exercise provide a basis for guidance on future large-scale simulation experiments and demonstrate the need for strategic investments in the evaluation and development of stock assessment methods",
author = "Deroba, {J. J.} and Butterworth, {D. S.} and Methot, {R. D.} and {De Oliveira}, {J. A. A.} and C. Fernandez and Anders Nielsen and Cadrin, {S. X.} and M. Dickey-Collas and Legault, {C. M.} and J. Ianelli and Valero, {J. L.} and Needle, {C. L.} and O'Malley, {J. M.} and Y.-J. Chang and Thompson, {G. G.} and C. Canales and Swain, {D. P.} and Miller, {D. C. M.} and Hintzen, {N. T.} and M. Bertignac and L. Ibaibarriaga and A. Silva and A. Murta and Kell, {L. T.} and {de Moor}, {C. L.} and Parma, {A. M.} and Dichmont, {C. M.} and Restrepo, {V. R.} and Y. Ye and E. Jardim and Spencer, {P. D.} and Hanselman, {D. H.} and J. Blaylock and M. Mood and Hulson, {P.- J. F.}",
year = "2015",
doi = "10.1093/icesjms/fst237",
language = "English",
volume = "72",
pages = "19--30",
journal = "I C E S Journal of Marine Science",
issn = "1054-3139",
publisher = "Oxford University Press",
number = "1",

}

Deroba, JJ, Butterworth, DS, Methot, RD, De Oliveira, JAA, Fernandez, C, Nielsen, A, Cadrin, SX, Dickey-Collas, M, Legault, CM, Ianelli, J, Valero, JL, Needle, CL, O'Malley, JM, Chang, Y-J, Thompson, GG, Canales, C, Swain, DP, Miller, DCM, Hintzen, NT, Bertignac, M, Ibaibarriaga, L, Silva, A, Murta, A, Kell, LT, de Moor, CL, Parma, AM, Dichmont, CM, Restrepo, VR, Ye, Y, Jardim, E, Spencer, PD, Hanselman, DH, Blaylock, J, Mood, M & Hulson, PJF 2015, 'Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods', ICES Journal of Marine Science, vol. 72, no. 1, pp. 19-30. https://doi.org/10.1093/icesjms/fst237

Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods. / Deroba, J. J.; Butterworth, D. S.; Methot, R. D.; De Oliveira, J. A. A.; Fernandez, C.; Nielsen, Anders; Cadrin, S. X.; Dickey-Collas, M.; Legault, C. M.; Ianelli, J.; Valero, J. L.; Needle, C. L.; O'Malley, J. M.; Chang, Y.-J.; Thompson, G. G.; Canales, C.; Swain, D. P.; Miller, D. C. M.; Hintzen, N. T.; Bertignac, M.; Ibaibarriaga, L.; Silva, A.; Murta, A.; Kell, L. T.; de Moor, C. L.; Parma, A. M.; Dichmont, C. M.; Restrepo, V. R.; Ye, Y.; Jardim, E.; Spencer, P. D.; Hanselman, D. H.; Blaylock, J.; Mood, M.; Hulson, P.- J. F.

In: ICES Journal of Marine Science, Vol. 72, No. 1, 2015, p. 19-30.

Research output: Contribution to journalJournal articleResearchpeer-review

TY - JOUR

T1 - Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods

AU - Deroba, J. J.

AU - Butterworth, D. S.

AU - Methot, R. D.

AU - De Oliveira, J. A. A.

AU - Fernandez, C.

AU - Nielsen, Anders

AU - Cadrin, S. X.

AU - Dickey-Collas, M.

AU - Legault, C. M.

AU - Ianelli, J.

AU - Valero, J. L.

AU - Needle, C. L.

AU - O'Malley, J. M.

AU - Chang, Y.-J.

AU - Thompson, G. G.

AU - Canales, C.

AU - Swain, D. P.

AU - Miller, D. C. M.

AU - Hintzen, N. T.

AU - Bertignac, M.

AU - Ibaibarriaga, L.

AU - Silva, A.

AU - Murta, A.

AU - Kell, L. T.

AU - de Moor, C. L.

AU - Parma, A. M.

AU - Dichmont, C. M.

AU - Restrepo, V. R.

AU - Ye, Y.

AU - Jardim, E.

AU - Spencer, P. D.

AU - Hanselman, D. H.

AU - Blaylock, J.

AU - Mood, M.

AU - Hulson, P.- J. F.

PY - 2015

Y1 - 2015

N2 - The World Conference on Stock Assessment Methods (July 2013) included a workshop on testing assessment methods through simulations. The exercise was made up of two steps applied to datasets from 14 representative fish stocks from around the world. Step 1 involved applying stock assessments to datasets with varying degrees of effort dedicated to optimizing fit. Step 2 was applied to a subset of the stocks and involved characteristics of given model fits being used to generate pseudo-data with error. These pseudo-data were then provided to assessment modellers and fits to the pseudo-data provided consistency checks within (self-tests) and among (cross-tests) assessmentmodels. Although trends in biomass were often similar across models, the scaling of absolute biomass was not consistent across models. Similar types of models tended to perform similarly (e.g. age based or production models). Self-testing and cross-testing of models are a useful diagnostic approach, and suggested that estimates in the most recent years of time-series were the least robust. Results from the simulation exercise provide a basis for guidance on future large-scale simulation experiments and demonstrate the need for strategic investments in the evaluation and development of stock assessment methods

AB - The World Conference on Stock Assessment Methods (July 2013) included a workshop on testing assessment methods through simulations. The exercise was made up of two steps applied to datasets from 14 representative fish stocks from around the world. Step 1 involved applying stock assessments to datasets with varying degrees of effort dedicated to optimizing fit. Step 2 was applied to a subset of the stocks and involved characteristics of given model fits being used to generate pseudo-data with error. These pseudo-data were then provided to assessment modellers and fits to the pseudo-data provided consistency checks within (self-tests) and among (cross-tests) assessmentmodels. Although trends in biomass were often similar across models, the scaling of absolute biomass was not consistent across models. Similar types of models tended to perform similarly (e.g. age based or production models). Self-testing and cross-testing of models are a useful diagnostic approach, and suggested that estimates in the most recent years of time-series were the least robust. Results from the simulation exercise provide a basis for guidance on future large-scale simulation experiments and demonstrate the need for strategic investments in the evaluation and development of stock assessment methods

U2 - 10.1093/icesjms/fst237

DO - 10.1093/icesjms/fst237

M3 - Journal article

VL - 72

SP - 19

EP - 30

JO - I C E S Journal of Marine Science

JF - I C E S Journal of Marine Science

SN - 1054-3139

IS - 1

ER -