Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study

Joakim Edin, Lasse Borgholt, Alexander Junge, Maria Maistro, Jakob D. Havtorn, Tuukka Ruotsalo, Lars Maaløe

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Abstract

Medical coding is the task of assigning medical codes to clinical free-text documentation. Healthcare professionals manually assign such codes to track patient diagnoses and treatments. Automated medical coding can considerably alleviate this administrative burden. In this paper, we reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. In previous work, the macro F1 score has been calculated sub-optimally, and our correction doubles it. We contribute a revised model comparison using stratified sampling and identical experimental setups, including hyperparameters and decision boundary tuning. We analyze prediction errors to validate and falsify assumptions of previous works. The analysis confirms that all models struggle with rare codes, while long documents only have a negligible impact. Finally, we present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models. We release our code, model parameters, and new MIMIC-III and MIMIC-IV training and evaluation pipelines to accommodate fair future comparisons.

Original languageEnglish
Title of host publicationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
PublisherAssociation for Computing Machinery
Publication date2023
Pages2572-2582
ISBN (Electronic)978-1-4503-9408-6
DOIs
Publication statusPublished - 2023
Event46th International ACM SIGIR Conference on Research and Development in Information Retrieval - Taipei, Taiwan, Province of China
Duration: 23 Jul 202327 Jul 2023

Conference

Conference46th International ACM SIGIR Conference on Research and Development in Information Retrieval
Country/TerritoryTaiwan, Province of China
CityTaipei
Period23/07/202327/07/2023
SponsorAssociation for Computing Machinery

Keywords

  • Automated Medical Coding
  • MIMIC
  • Reproducibility

Fingerprint

Dive into the research topics of 'Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study'. Together they form a unique fingerprint.

Cite this