Critical Assessment of Metagenome Interpretation: the second round of challenges

Fernando Meyer; Adrian Fritz; Zhi-Luo Deng; David Koslicki; Till Robin Lesker; Alexey Gurevich; Gary Robertson; Mohammed Alser; Dmitry Antipov; Francesco Beghini; Denis Bertrand; Jaqueline J. Brito; C. Titus Brown; Jan Buchmann; Aydin Buluç; Bo Chen; Rayan Chikhi; Philip T.L.C. Clausen; Alexandru Cristian; Piotr Wojciech Dabrowski; Aaron E. Darling; Rob Egan; Eleazar Eskin; Evangelos Georganas; Eugene Goltsman; Melissa A. Gray; Lars Hestbjerg Hansen; Steven Hofmeyr; Pingqin Huang; Luiz Irber; Huijue Jia; Tue Sparholt Jørgensen; Silas D. Kieser; Terje Klemetsen; Axel Kola; Mikhail Kolmogorov; Anton Korobeynikov; Jason Kwan; Nathan LaPierre; Claire Lemaitre; Chenhao Li; Antoine Limasset; Fabio Malcher-Miranda; Serghei Mangul; Vanessa R. Marcelino; Camille Marchet; Pierre Marijon; Dmitry Meleshko; Daniel R. Mende; Alessio Milanese; Niranjan Nagarajan; Jakob Nissen; Sergey Nurk; Leonid Oliker; Lucas Paoli; Pierre Peterlongo; Vitor C. Piro; Jacob S. Porter; Simon Rasmussen; Evan R. Rees; Knut Reinert; Bernhard Renard; Espen Mikal Robertsen; Gail L. Rosen; Hans-Joachim Ruscheweyh; Varuni Sarwal; Nicola Segata; Enrico Seiler; Lizhen Shi; Fengzhu Sun; Shinichi Sunagawa; Søren Johannes Sørensen; Ashleigh Thomas; Chengxuan Tong; Mirko Trajkovski; Julien Tremblay; Gherman Uritskiy; Riccardo Vicedomini; Zhengyang Wang; Ziye Wang; Zhong Wang; Andrew Warren; Nils Peder Willassen; Katherine Yelick; Ronghui You; Georg Zeller; Zhengqiao Zhao; Shanfeng Zhu; Jie Zhu; Ruben Garrido-Oter; Petra Gastmeier; Stephane Hacquard; Susanne Häußler; Ariane Khaledi; Friederike Maechler; Fantin Mesny; Simona Radutoiu; Paul Schulze-Lefert; Nathiana Smit; Till Strowig; Andreas Bremges; Alexander Sczyrba; Alice Carolyn McHardy

doi:10.1038/s41592-022-01431-4

Critical Assessment of Metagenome Interpretation: the second round of challenges

Fernando Meyer, Adrian Fritz, Zhi-Luo Deng, David Koslicki, Till Robin Lesker, Alexey Gurevich, Gary Robertson, Mohammed Alser, Dmitry Antipov, Francesco Beghini, Denis Bertrand, Jaqueline J. Brito, C. Titus Brown, Jan Buchmann, Aydin Buluç, Bo Chen, Rayan Chikhi, Philip T.L.C. Clausen, Alexandru Cristian, Piotr Wojciech DabrowskiAaron E. Darling, Rob Egan, Eleazar Eskin, Evangelos Georganas, Eugene Goltsman, Melissa A. Gray, Lars Hestbjerg Hansen, Steven Hofmeyr, Pingqin Huang, Luiz Irber, Huijue Jia, Tue Sparholt Jørgensen, Silas D. Kieser, Terje Klemetsen, Axel Kola, Mikhail Kolmogorov, Anton Korobeynikov, Jason Kwan, Nathan LaPierre, Claire Lemaitre, Chenhao Li, Antoine Limasset, Fabio Malcher-Miranda, Serghei Mangul, Vanessa R. Marcelino, Camille Marchet, Pierre Marijon, Dmitry Meleshko, Daniel R. Mende, Alessio Milanese, Niranjan Nagarajan, Jakob Nissen, Sergey Nurk, Leonid Oliker, Lucas Paoli, Pierre Peterlongo, Vitor C. Piro, Jacob S. Porter, Simon Rasmussen, Evan R. Rees, Knut Reinert, Bernhard Renard, Espen Mikal Robertsen, Gail L. Rosen, Hans-Joachim Ruscheweyh, Varuni Sarwal, Nicola Segata, Enrico Seiler, Lizhen Shi, Fengzhu Sun, Shinichi Sunagawa, Søren Johannes Sørensen, Ashleigh Thomas, Chengxuan Tong, Mirko Trajkovski, Julien Tremblay, Gherman Uritskiy, Riccardo Vicedomini, Zhengyang Wang, Ziye Wang, Zhong Wang, Andrew Warren, Nils Peder Willassen, Katherine Yelick, Ronghui You, Georg Zeller, Zhengqiao Zhao, Shanfeng Zhu, Jie Zhu, Ruben Garrido-Oter, Petra Gastmeier, Stephane Hacquard, Susanne Häußler, Ariane Khaledi, Friederike Maechler, Fantin Mesny, Simona Radutoiu, Paul Schulze-Lefert, Nathiana Smit, Till Strowig, Andreas Bremges, Alexander Sczyrba, Alice Carolyn McHardy^*

^*Corresponding author for this work

Research output: Contribution to journal › Journal article › Research › peer-review

99 Downloads (Pure)

Abstract

Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.

Original language	English
Journal	Nature Methods
Volume	19
Issue number	4
Pages (from-to)	429-440
Number of pages	12
ISSN	1548-7091
DOIs	https://doi.org/10.1038/s41592-022-01431-4
Publication status	Published - 2022

Access to Document

10.1038/s41592-022-01431-4Licence: CC BY

FulltextFinal published version, 2.23 MBLicence: CC BY

OpenUrl availability

Full text

Cite this

Meyer, F., Fritz, A., Deng, Z-L., Koslicki, D., Lesker, T. R., Gurevich, A., Robertson, G., Alser, M., Antipov, D., Beghini, F., Bertrand, D., Brito, J. J., Brown, C. T., Buchmann, J., Buluç, A., Chen, B., Chikhi, R., Clausen, P. T. L. C., Cristian, A., ... McHardy, A. C. (2022). Critical Assessment of Metagenome Interpretation: the second round of challenges. Nature Methods, 19(4), 429-440. https://doi.org/10.1038/s41592-022-01431-4

@article{361a7860954b4152ab611ab9ccd4284e,

title = "Critical Assessment of Metagenome Interpretation: the second round of challenges",

abstract = "Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.",

author = "Fernando Meyer and Adrian Fritz and Zhi-Luo Deng and David Koslicki and Lesker, {Till Robin} and Alexey Gurevich and Gary Robertson and Mohammed Alser and Dmitry Antipov and Francesco Beghini and Denis Bertrand and Brito, {Jaqueline J.} and Brown, {C. Titus} and Jan Buchmann and Aydin Bulu{\c c} and Bo Chen and Rayan Chikhi and Clausen, {Philip T.L.C.} and Alexandru Cristian and Dabrowski, {Piotr Wojciech} and Darling, {Aaron E.} and Rob Egan and Eleazar Eskin and Evangelos Georganas and Eugene Goltsman and Gray, {Melissa A.} and Hansen, {Lars Hestbjerg} and Steven Hofmeyr and Pingqin Huang and Luiz Irber and Huijue Jia and J{\o}rgensen, {Tue Sparholt} and Kieser, {Silas D.} and Terje Klemetsen and Axel Kola and Mikhail Kolmogorov and Anton Korobeynikov and Jason Kwan and Nathan LaPierre and Claire Lemaitre and Chenhao Li and Antoine Limasset and Fabio Malcher-Miranda and Serghei Mangul and Marcelino, {Vanessa R.} and Camille Marchet and Pierre Marijon and Dmitry Meleshko and Mende, {Daniel R.} and Alessio Milanese and Niranjan Nagarajan and Jakob Nissen and Sergey Nurk and Leonid Oliker and Lucas Paoli and Pierre Peterlongo and Piro, {Vitor C.} and Porter, {Jacob S.} and Simon Rasmussen and Rees, {Evan R.} and Knut Reinert and Bernhard Renard and Robertsen, {Espen Mikal} and Rosen, {Gail L.} and Hans-Joachim Ruscheweyh and Varuni Sarwal and Nicola Segata and Enrico Seiler and Lizhen Shi and Fengzhu Sun and Shinichi Sunagawa and S{\o}rensen, {S{\o}ren Johannes} and Ashleigh Thomas and Chengxuan Tong and Mirko Trajkovski and Julien Tremblay and Gherman Uritskiy and Riccardo Vicedomini and Zhengyang Wang and Ziye Wang and Zhong Wang and Andrew Warren and Willassen, {Nils Peder} and Katherine Yelick and Ronghui You and Georg Zeller and Zhengqiao Zhao and Shanfeng Zhu and Jie Zhu and Ruben Garrido-Oter and Petra Gastmeier and Stephane Hacquard and Susanne H{\"a}u{\ss}ler and Ariane Khaledi and Friederike Maechler and Fantin Mesny and Simona Radutoiu and Paul Schulze-Lefert and Nathiana Smit and Till Strowig and Andreas Bremges and Alexander Sczyrba and McHardy, {Alice Carolyn}",

year = "2022",

doi = "10.1038/s41592-022-01431-4",

language = "English",

volume = "19",

pages = "429--440",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Nature Publishing Group",

number = "4",

}

Meyer, F, Fritz, A, Deng, Z-L, Koslicki, D, Lesker, TR, Gurevich, A, Robertson, G, Alser, M, Antipov, D, Beghini, F, Bertrand, D, Brito, JJ, Brown, CT, Buchmann, J, Buluç, A, Chen, B, Chikhi, R, Clausen, PTLC, Cristian, A, Dabrowski, PW, Darling, AE, Egan, R, Eskin, E, Georganas, E, Goltsman, E, Gray, MA, Hansen, LH, Hofmeyr, S, Huang, P, Irber, L, Jia, H, Jørgensen, TS, Kieser, SD, Klemetsen, T, Kola, A, Kolmogorov, M, Korobeynikov, A, Kwan, J, LaPierre, N, Lemaitre, C, Li, C, Limasset, A, Malcher-Miranda, F, Mangul, S, Marcelino, VR, Marchet, C, Marijon, P, Meleshko, D, Mende, DR, Milanese, A, Nagarajan, N, Nissen, J, Nurk, S, Oliker, L, Paoli, L, Peterlongo, P, Piro, VC, Porter, JS, Rasmussen, S, Rees, ER, Reinert, K, Renard, B, Robertsen, EM, Rosen, GL, Ruscheweyh, H-J, Sarwal, V, Segata, N, Seiler, E, Shi, L, Sun, F, Sunagawa, S, Sørensen, SJ, Thomas, A, Tong, C, Trajkovski, M, Tremblay, J, Uritskiy, G, Vicedomini, R, Wang, Z, Wang, Z, Wang, Z, Warren, A, Willassen, NP, Yelick, K, You, R, Zeller, G, Zhao, Z, Zhu, S, Zhu, J, Garrido-Oter, R, Gastmeier, P, Hacquard, S, Häußler, S, Khaledi, A, Maechler, F, Mesny, F, Radutoiu, S, Schulze-Lefert, P, Smit, N, Strowig, T, Bremges, A, Sczyrba, A & McHardy, AC 2022, 'Critical Assessment of Metagenome Interpretation: the second round of challenges', Nature Methods, vol. 19, no. 4, pp. 429-440. https://doi.org/10.1038/s41592-022-01431-4

TY - JOUR

T1 - Critical Assessment of Metagenome Interpretation

T2 - the second round of challenges

AU - Meyer, Fernando

AU - Fritz, Adrian

AU - Deng, Zhi-Luo

AU - Koslicki, David

AU - Lesker, Till Robin

AU - Gurevich, Alexey

AU - Robertson, Gary

AU - Alser, Mohammed

AU - Antipov, Dmitry

AU - Beghini, Francesco

AU - Bertrand, Denis

AU - Brito, Jaqueline J.

AU - Brown, C. Titus

AU - Buchmann, Jan

AU - Buluç, Aydin

AU - Chen, Bo

AU - Chikhi, Rayan

AU - Clausen, Philip T.L.C.

AU - Cristian, Alexandru

AU - Dabrowski, Piotr Wojciech

AU - Darling, Aaron E.

AU - Egan, Rob

AU - Eskin, Eleazar

AU - Georganas, Evangelos

AU - Goltsman, Eugene

AU - Gray, Melissa A.

AU - Hansen, Lars Hestbjerg

AU - Hofmeyr, Steven

AU - Huang, Pingqin

AU - Irber, Luiz

AU - Jia, Huijue

AU - Jørgensen, Tue Sparholt

AU - Kieser, Silas D.

AU - Klemetsen, Terje

AU - Kola, Axel

AU - Kolmogorov, Mikhail

AU - Korobeynikov, Anton

AU - Kwan, Jason

AU - LaPierre, Nathan

AU - Lemaitre, Claire

AU - Li, Chenhao

AU - Limasset, Antoine

AU - Malcher-Miranda, Fabio

AU - Mangul, Serghei

AU - Marcelino, Vanessa R.

AU - Marchet, Camille

AU - Marijon, Pierre

AU - Meleshko, Dmitry

AU - Mende, Daniel R.

AU - Milanese, Alessio

AU - Nagarajan, Niranjan

AU - Nissen, Jakob

AU - Nurk, Sergey

AU - Oliker, Leonid

AU - Paoli, Lucas

AU - Peterlongo, Pierre

AU - Piro, Vitor C.

AU - Porter, Jacob S.

AU - Rasmussen, Simon

AU - Rees, Evan R.

AU - Reinert, Knut

AU - Renard, Bernhard

AU - Robertsen, Espen Mikal

AU - Rosen, Gail L.

AU - Ruscheweyh, Hans-Joachim

AU - Sarwal, Varuni

AU - Segata, Nicola

AU - Seiler, Enrico

AU - Shi, Lizhen

AU - Sun, Fengzhu

AU - Sunagawa, Shinichi

AU - Sørensen, Søren Johannes

AU - Thomas, Ashleigh

AU - Tong, Chengxuan

AU - Trajkovski, Mirko

AU - Tremblay, Julien

AU - Uritskiy, Gherman

AU - Vicedomini, Riccardo

AU - Wang, Zhengyang

AU - Wang, Ziye

AU - Wang, Zhong

AU - Warren, Andrew

AU - Willassen, Nils Peder

AU - Yelick, Katherine

AU - You, Ronghui

AU - Zeller, Georg

AU - Zhao, Zhengqiao

AU - Zhu, Shanfeng

AU - Zhu, Jie

AU - Garrido-Oter, Ruben

AU - Gastmeier, Petra

AU - Hacquard, Stephane

AU - Häußler, Susanne

AU - Khaledi, Ariane

AU - Maechler, Friederike

AU - Mesny, Fantin

AU - Radutoiu, Simona

AU - Schulze-Lefert, Paul

AU - Smit, Nathiana

AU - Strowig, Till

AU - Bremges, Andreas

AU - Sczyrba, Alexander

AU - McHardy, Alice Carolyn

PY - 2022

Y1 - 2022

N2 - Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.

AB - Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.

U2 - 10.1038/s41592-022-01431-4

DO - 10.1038/s41592-022-01431-4

M3 - Journal article

C2 - 35396482

SN - 1548-7091

VL - 19

SP - 429

EP - 440

JO - Nature Methods

JF - Nature Methods

IS - 4

ER -

Critical Assessment of Metagenome Interpretation: the second round of challenges

Abstract

Access to Document

OpenUrl availability

Fingerprint

Cite this