Understanding metric-related pitfalls in image analysis validation · SciRah

article Open access

Understanding metric-related pitfalls in image analysis validation

Annika Reinke^{a b c d}✉, Minu D. Tizabi^{d e f}✉, Michael Baumgartner^g, Matthias Eisenmann^d, Doreen Heckmann-Nötzel^{d h}, Ali Emre Kavur^{d g i}, Tim Rädsch^{d j}, Carole H. Sudre^{k l}, Laura Ación^{m n}, Michela Antonelli^{l o}, Tal Arbel^p

Nature Methods Nature Portfolio2024-02-01 245 citations36 references

Looking for the full paper…

Export

Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.

Abstract

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation. This Perspective presents a reliable and comprehensive source of information on pitfalls related to validation metrics in image analysis, with an emphasis on biomedical imaging.

TL;DRThe present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis and serves to enhance global comprehension of a key topic in image analysis validation. (Semantic Scholar)

Plain-language summary

AI illustration of the topic of “Understanding metric-related pitfalls in image analysis validation”

Discussion

^aFaculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany. a.reinke@dkfz-heidelberg.de
^bGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. a.reinke@dkfz-heidelberg.de
^cGerman Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany. a.reinke@dkfz-heidelberg.de
^dGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
^eGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. m.tizabi@dkfz-heidelberg.de
^fNational Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany. m.tizabi@dkfz-heidelberg.de
^gGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
^hNational Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
ⁱGerman Cancer Research Center (DKFZ) Heidelberg, HI Applied Computer Vision Lab, Heidelberg, Germany
^jGerman Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany
^kMRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK
^lSchool of Biomedical Engineering and Imaging Science, King's College London, London, UK
^mInstituto de Cálculo, CONICET - Universidad de Buenos Aires, Buenos Aires, Argentina
ⁿInstituto de Cálculo, CONICET – Universidad de Buenos Aires, Buenos Aires, Argentina
^oCentre for Medical Image Computing, University College London, London, UK
^pCentre for Intelligent Machines and MILA (Quebec Artificial Intelligence Institute), McGill University, Montréal, Quebec, Canada
^qCenter for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
^rDivision of Computational Pathology, Dept of Pathology & Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, USA
^sDepartment of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel
^tEuropean Federation for Medical Informatics, Le Mont-sur-Lausanne, Switzerland
^uFrankfurt Cancer Insititute, Frankfurt am Main, Germany
^vGerman Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany
^wGerman Cancer Research Center (DKFZ) Heidelberg, Heidelberg, Germany
^xGoethe University Frankfurt, Department of Informatics, Frankfurt am Main, Germany
^yGoethe University Frankfurt, Department of Medicine, Frankfurt am Main, Germany
^zDepartment of Computer Science, IT University of Copenhagen, Copenhagen, Denmark
^aaLeibniz-Institut für Analytische Wissenschaften - ISAS - e.V., Dortmund, Germany
^abLeibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany
^acImaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA
^adCenter for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA
^aeInstituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Autónoma de Buenos Aires, Buenos Aires, Argentina
^afUniversitat Pompeu Fabra, Barcelona, Spain
^agUniversity of Adelaide, Adelaide, South Australia, Australia
^ahFraunhofer MEVIS, Bremen, Germany
^aiRadboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, the Netherlands
^ajDepartment of Computing, Imperial College London, South Kensington Campus, London, UK
^akFaculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
^alDepartment of Surgery, Perelman School of Medicine, Philadelphia, PA, USA
^amGeneral Robotics Automation Sensing and Perception Laboratory, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA
^anDepartment of Computer Science, University of Toronto, Toronto, Ontario, Canada
^aoDepartment of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada
^apPrincess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
^aqVector Institute for Artificial Intelligence, Toronto, Ontario, Canada
^arDepartment of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
^asINSERM, Paris, France
^atLaboratoire Traitement du Signal et de l'Image - UMR_S 1099, Université de Rennes 1, Rennes, France
^auLaboratoire Traitement du Signal et de l'Image – UMR_S 1099, Université de Rennes 1, Rennes, France
✉Corresponding author