From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge

Research output: Research - peer-reviewJournal article – Annual report year: 2018

DOI

  • Author: Bandi, Peter

    Radboud University Nijmegen, Netherlands

  • Author: Geessink, Oscar

    Radboud University Nijmegen, Netherlands

  • Author: Manson, Quirine

    University Medical Centre Utrecht, Netherlands

  • Author: van Dijk, Marcory

    Rijnstate Hospital, Netherlands

  • Author: Balkenhol, Maschenka

    Radboud University Nijmegen, Netherlands

  • Author: Hermsen, Meyke

    Radboud University Nijmegen, Netherlands

  • Author: Bejnordi, Babak Ehteshami

    Radboud University Nijmegen, Netherlands

  • Author: Lee, Byungjae

    Lunit Inc., Korea, Republic of

  • Author: Paeng, Kyunghyun

    Lunit Inc., Korea, Republic of

  • Author: Zhong, Aoxiao

    Harvard Medical School, United States

  • Author: Li, Quanzheng

    Harvard Medical School, United States

  • Author: Zanjani, Farhad Ghazvinian

    Eindhoven University of Technology, Netherlands

  • Author: Zinger, Svitlana

    Eindhoven University of Technology, Netherlands

  • Author: Fukuta, Keisuke

    University of Tokyo, Japan

  • Author: Komura, Daisuke

    Tokyo Medical and Dental University, Japan

  • Author: Ovtcharov, Vlado

    Indica Labs, United States

  • Author: Cheng, Shenghua

    Huazhong University of Science and Technology, China

  • Author: Zeng, Shaoqun

    Huazhong University of Science and Technology, China

  • Author: Thagaard, Jeppe

    Image Analysis & Computer Graphics, Department of Applied Mathematics and Computer Science , Technical University of Denmark, Richard Petersens Plads, 2800, Kgs. Lyngby, Denmark

  • Author: Dahl, Anders Bjorholm

    Image Analysis & Computer Graphics, Department of Applied Mathematics and Computer Science , Technical University of Denmark, Richard Petersens Plads, 2800, Kgs. Lyngby, Denmark

  • Author: Lin, Huangjing

    Chinese University of Hong Kong, Hong Kong

  • Author: Chen, Hao

    Chinese University of Hong Kong, Hong Kong

  • Author: Jacobsson, Ludwig

    ContextVision AB, Sweden

  • Author: Hedlund, Martin

    ContextVision AB, Sweden

  • Author: Cetin, Melih

    Middle East Technical University (METU), Turkey

  • Author: Halici, Eren

    Middle East Technical University (METU), Turkey

  • Author: Jackson, Hunter

    Proscia Inc., United States

  • Author: Chen, Richard

    Proscia Inc., United States

  • Author: Both, Fabian

    Karlsruhe Institute of Technology

  • Author: Franke, Jorg

    Karlsruhe Institute of Technology

  • Author: Kusters-Vandevelde, Heidi

    Canisius Wilhelmina Hospital, Netherlands

  • Author: Vreuls, Willem

    Canisius Wilhelmina Hospital, Netherlands

  • Author: Bult, Peter

    Radboud University Medical Centre, Netherlands

  • Author: van Ginneken, Bram

    Radboud University Medical Centre, Netherlands

  • Author: van der Laak, Jeroen

    Radboud University Medical Centre, Netherlands

  • Author: Litjens, Geert

    Radboud University Medical Centre, Netherlands

View graph of relations

Automated detection of cancer metastases in lymph nodes has the potential to improve assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 conference in Melbourne. Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 whole-slide images. The evaluation metric used was a quadratic weighted Cohen’s kappa. We discuss the algorithmic details of the ten best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre-and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy and pre-and postprocessing methodology. Overall, the kappa metric ranged from 0.89 to -0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%. Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targets for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.
Original languageEnglish
JournalI E E E Transactions on Medical Imaging
Issue number11
Number of pages11
ISSN0278-0062
DOIs
StateAccepted/In press - 2018
CitationsWeb of Science® Times Cited: No match on DOI

    Research areas

  • Lymph nodes, Biomedical imaging, Tumors, Metastasis, Pathology, Hospitals, Breast cancer, Sentinel lymph node, Lymph node metastases, Whole-slide images, Grand challenge
Download as:
Download as PDF
Select render style:
APAAuthorCBE/CSEHarvardMLAStandardVancouverShortLong
PDF
Download as HTML
Select render style:
APAAuthorCBE/CSEHarvardMLAStandardVancouverShortLong
HTML
Download as Word
Select render style:
APAAuthorCBE/CSEHarvardMLAStandardVancouverShortLong
Word

ID: 154611046