Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging.

Convolutional Neural Network (CNN) Feature Detection Technical Aspects Technology Assessment

Journal

Radiology. Artificial intelligence
ISSN: 2638-6100
Titre abrégé: Radiol Artif Intell
Pays: United States
ID NLM: 101746556

Informations de publication

Date de publication:
Nov 2021
Historique:
received: 17 11 2020
revised: 13 09 2021
accepted: 20 09 2021
entrez: 6 12 2021
pubmed: 7 12 2021
medline: 7 12 2021
Statut: epublish

Résumé

To evaluate the trustworthiness of saliency maps for abnormality localization in medical imaging. Using two large publicly available radiology datasets (Society for Imaging Informatics in Medicine-American College of Radiology Pneumothorax Segmentation dataset and Radiological Society of North America Pneumonia Detection Challenge dataset), the performance of eight commonly used saliency map techniques were quantified in regard to All eight saliency map techniques failed at least one of the criteria and were inferior in performance compared with localization networks. For pneumothorax segmentation, the AUPRC ranged from 0.024 to 0.224, while a U-Net achieved a significantly superior AUPRC of 0.404 ( The use of saliency maps in the high-risk domain of medical imaging warrants additional scrutiny and recommend that detection or segmentation models be used if localization is the desired output of the network.

Identifiants

DOI: 10.1148/ryai.2021200267 PMID: 34870212 PMC: PMC8637231
pubmed: 34870212
doi: 10.1148/ryai.2021200267
pmc: PMC8637231
doi:

Types de publication

Journal Article

Langues

eng

Pagination

e200267

Informations de copyright

2021 by the Radiological Society of North America, Inc.

Déclaration de conflit d'intérêts

Disclosures of Conflicts of Interest: N.A. No relevant relationships. N.G. No relevant relationships. P.S. No relevant relationships. K.C. Research support through training grant from National Institute of Biomedical Imaging and Bioengineering of National Institutes of Health (NIH) (award no. 5T32EB1680) and by National Cancer Institute of NIH (award no. F30CA239407). M.A. No relevant relationships. B.C. No relevant relationships. K.H. No relevant relationships. S.G. No relevant relationships. J.P. No relevant relationships. M.G. No relevant relationships. J.A. No relevant relationships. M.D.L. Member of Radiology: Artificial Intelligence trainee editorial board; Radiological Society of North America Research and Education Foundation Presidents Circle grant. J.K.C. Institution has grant/grants pending from NIH, GE, Genentech, National Science Foundation; deputy editor for Radiology: Artificial Intelligence; payment from IBM for lectures including service on speakers bureaus; royalties from Boston AI.

Références

PLoS Med. 2018 Nov 27;15(11):e1002699
pubmed: 30481176
J Clin Epidemiol. 2015 Aug;68(8):855-9
pubmed: 25881487
Nat Biomed Eng. 2020 Jan;4(1):18-27
pubmed: 31873211
J Med Syst. 2019 Jul 20;43(9):289
pubmed: 31327058
PLoS Med. 2018 Nov 20;15(11):e1002686
pubmed: 30457988
NPJ Digit Med. 2020 Mar 26;3:48
pubmed: 32258430
Ophthalmology. 2019 Apr;126(4):552-564
pubmed: 30553900
Pattern Anal Appl. 2021 May 9;:1-14
pubmed: 33994847
Radiol Artif Intell. 2020 Jul 22;2(4):e200079
pubmed: 33928256
Clin Ophthalmol. 2021 Jun 18;15:2573-2581
pubmed: 34177258
J Cancer Res Clin Oncol. 2019 Apr;145(4):829-837
pubmed: 30603908
Proc ACM Conf Health Inference Learn (2020). 2020 Apr;2020:151-159
pubmed: 33196064
IEEE J Biomed Health Inform. 2021 May;25(5):1660-1672
pubmed: 32956067
Lancet Digit Health. 2019 Jul;1(3):e136-e147
pubmed: 31448366
J Med Imaging (Bellingham). 2017 Jul;4(3):035501
pubmed: 28924574
J Am Coll Radiol. 2020 Dec;17(12):1653-1662
pubmed: 32592660
Radiol Artif Intell. 2019 Jan 30;1(1):e180041
pubmed: 33937785

Auteurs

Nishanth Arun (N)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0003-1424-4703

Nathan Gaw (N)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0001-7879-8946

Praveer Singh (P)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0001-6641-2030

Ken Chang (K)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0001-6956-5059

Mehak Aggarwal (M)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).

Bryan Chen (B)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).

Katharina Hoebel (K)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0002-1881-7065

Sharut Gupta (S)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0003-1848-9935

Jay Patel (J)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0002-8507-795X

Mishka Gidwani (M)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0002-8477-2539

Julius Adebayo (J)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).

Matthew D Li (MD)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0002-6207-4940

Jayashree Kalpathy-Cramer (J)

Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 149 13th St, Boston, MA 02129 (N.A., P.S., K.C., M.A., B.C., K.H., S.G., J.P., M.G., M.D.L., J.K.C.); Department of Computer Science, Shiv Nadar University, Greater Noida, India (N.A.); Department of Operational Sciences, Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, Dayton, Ohio (N.G.); and Massachusetts Institute of Technology, Cambridge, Mass (K.C., B.C., K.H., J.P., J.A.).
ORCID: 0000-0001-8906-9618

Classifications MeSH