The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Bjoern H. Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Koen Van Leemput

Research output: Contribution to journalJournal articleResearchpeer-review

2792 Downloads (Pure)

Abstract

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients – manually annotated by up to four raters – and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all subregions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.
Original languageEnglish
JournalI E E E Transactions on Medical Imaging
Volume34
Issue number10
Pages (from-to)1993 - 2024
Number of pages33
ISSN0278-0062
DOIs
Publication statusPublished - 2015

Cite this

Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., ... Van Leemput, K. (2015). The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). I E E E Transactions on Medical Imaging, 34(10), 1993 - 2024 . https://doi.org/10.1109/TMI.2014.2377694
Menze, Bjoern H. ; Jakab, Andras ; Bauer, Stefan ; Kalpathy-Cramer, Jayashree ; Farahani, Keyvan ; Kirby, Justin ; Burren, Yuliya ; Porz, Nicole ; Slotboom, Johannes ; Wiest, Roland ; Van Leemput, Koen. / The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). In: I E E E Transactions on Medical Imaging. 2015 ; Vol. 34, No. 10. pp. 1993 - 2024 .
@article{4b589b6824a64a2a91e8e3b26cc0bf9e,
title = "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)",
abstract = "In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients – manually annotated by up to four raters – and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74-85{\%}), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all subregions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.",
author = "Menze, {Bjoern H.} and Andras Jakab and Stefan Bauer and Jayashree Kalpathy-Cramer and Keyvan Farahani and Justin Kirby and Yuliya Burren and Nicole Porz and Johannes Slotboom and Roland Wiest and {Van Leemput}, Koen",
year = "2015",
doi = "10.1109/TMI.2014.2377694",
language = "English",
volume = "34",
pages = "1993 -- 2024",
journal = "I E E E Transactions on Medical Imaging",
issn = "0278-0062",
publisher = "Institute of Electrical and Electronics Engineers",
number = "10",

}

Menze, BH, Jakab, A, Bauer, S, Kalpathy-Cramer, J, Farahani, K, Kirby, J, Burren, Y, Porz, N, Slotboom, J, Wiest, R & Van Leemput, K 2015, 'The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)', I E E E Transactions on Medical Imaging, vol. 34, no. 10, pp. 1993 - 2024 . https://doi.org/10.1109/TMI.2014.2377694

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). / Menze, Bjoern H.; Jakab, Andras ; Bauer, Stefan ; Kalpathy-Cramer, Jayashree ; Farahani, Keyvan ; Kirby, Justin; Burren, Yuliya ; Porz, Nicole; Slotboom, Johannes ; Wiest, Roland; Van Leemput, Koen.

In: I E E E Transactions on Medical Imaging, Vol. 34, No. 10, 2015, p. 1993 - 2024 .

Research output: Contribution to journalJournal articleResearchpeer-review

TY - JOUR

T1 - The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

AU - Menze, Bjoern H.

AU - Jakab, Andras

AU - Bauer, Stefan

AU - Kalpathy-Cramer, Jayashree

AU - Farahani, Keyvan

AU - Kirby, Justin

AU - Burren, Yuliya

AU - Porz, Nicole

AU - Slotboom, Johannes

AU - Wiest, Roland

AU - Van Leemput, Koen

PY - 2015

Y1 - 2015

N2 - In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients – manually annotated by up to four raters – and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all subregions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

AB - In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients – manually annotated by up to four raters – and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all subregions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

U2 - 10.1109/TMI.2014.2377694

DO - 10.1109/TMI.2014.2377694

M3 - Journal article

VL - 34

SP - 1993

EP - 2024

JO - I E E E Transactions on Medical Imaging

JF - I E E E Transactions on Medical Imaging

SN - 0278-0062

IS - 10

ER -

Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). I E E E Transactions on Medical Imaging. 2015;34(10):1993 - 2024 . https://doi.org/10.1109/TMI.2014.2377694