External Validation of a Neural Network Model in Aneurysmal Subarachnoid Hemorrhage: A Comparison With Conventional Logistic Regression Models.
Journal
Neurosurgery
ISSN: 1524-4040
Titre abrégé: Neurosurgery
Pays: United States
ID NLM: 7802914
Informations de publication
Date de publication:
01 05 2022
01 05 2022
Historique:
received:
01
06
2021
accepted:
10
11
2021
pubmed:
4
2
2022
medline:
21
4
2022
entrez:
3
2
2022
Statut:
ppublish
Résumé
Interest in machine learning (ML)-based predictive modeling has led to the development of models predicting outcomes after aneurysmal subarachnoid hemorrhage (aSAH), including the Nijmegen acute subarachnoid hemorrhage calculator (Nutshell). Generalizability of such models to external data remains unclear. To externally validate the performance of the Nutshell tool while comparing it with the conventional Subarachnoid Hemorrhage International Trialists (SAHIT) models and to review the ML literature on outcome prediction after aSAH and aneurysm treatment. A prospectively maintained database of patients with aSAH presenting consecutively to our institution in the 2013 to 2018 period was used. The web-based Nutshell and SAHIT calculators were used to derive the risks of poor long-term (12-18 months) outcomes and 30-day mortality. Discrimination was evaluated using the area under the curve (AUC), and calibration was investigated using calibration plots. The literature on relevant ML models was surveyed for a synopsis. In 269 patients with aSAH, the SAHIT models outperformed the Nutshell tool (AUC: 0.786 vs 0.689, P = .025) in predicting long-term functional outcomes. A logistic regression model of the Nutshell variables derived from our data achieved adequate discrimination (AUC = 0.759) of poor outcomes. The SAHIT models outperformed the Nutshell tool in predicting 30-day mortality (AUC: 0.810 vs 0.636, P < .001). Calibration properties were more favorable for the SAHIT models. Most published aneurysm-related ML-based outcome models lack external validation and usable testing platforms. The Nutshell tool demonstrated limited performance on external validation in comparison with the SAHIT models. External validation and the dissemination of testing platforms for ML models must be emphasized.
Sections du résumé
BACKGROUND
Interest in machine learning (ML)-based predictive modeling has led to the development of models predicting outcomes after aneurysmal subarachnoid hemorrhage (aSAH), including the Nijmegen acute subarachnoid hemorrhage calculator (Nutshell). Generalizability of such models to external data remains unclear.
OBJECTIVE
To externally validate the performance of the Nutshell tool while comparing it with the conventional Subarachnoid Hemorrhage International Trialists (SAHIT) models and to review the ML literature on outcome prediction after aSAH and aneurysm treatment.
METHODS
A prospectively maintained database of patients with aSAH presenting consecutively to our institution in the 2013 to 2018 period was used. The web-based Nutshell and SAHIT calculators were used to derive the risks of poor long-term (12-18 months) outcomes and 30-day mortality. Discrimination was evaluated using the area under the curve (AUC), and calibration was investigated using calibration plots. The literature on relevant ML models was surveyed for a synopsis.
RESULTS
In 269 patients with aSAH, the SAHIT models outperformed the Nutshell tool (AUC: 0.786 vs 0.689, P = .025) in predicting long-term functional outcomes. A logistic regression model of the Nutshell variables derived from our data achieved adequate discrimination (AUC = 0.759) of poor outcomes. The SAHIT models outperformed the Nutshell tool in predicting 30-day mortality (AUC: 0.810 vs 0.636, P < .001). Calibration properties were more favorable for the SAHIT models. Most published aneurysm-related ML-based outcome models lack external validation and usable testing platforms.
CONCLUSION
The Nutshell tool demonstrated limited performance on external validation in comparison with the SAHIT models. External validation and the dissemination of testing platforms for ML models must be emphasized.
Identifiants
pubmed: 35113076
doi: 10.1227/neu.0000000000001857
pii: 00006123-202205000-00009
doi:
Types de publication
Journal Article
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
552-561Informations de copyright
Copyright © Congress of Neurological Surgeons 2022. All rights reserved.
Références
Hop JW, Rinkel GJ, Algra A, van Gijn J. Case-fatality rates and functional outcome after subarachnoid hemorrhage: a systematic review. Stroke. 1997;28(3):660-664.
Huhtakangas J, Lehto H, Seppä K, et al. Long-term excess mortality after aneurysmal subarachnoid hemorrhage: patients with multiple aneurysms at risk. Stroke. 2015;46(7):1813-1818.
Jaja BNR, Saposnik G, Lingsma HF, et al. Development and validation of outcome prediction models for aneurysmal subarachnoid haemorrhage: the SAHIT multinational cohort study. BMJ. 2018;360:j5745.
Feghali J, Gami A, Rapaport S, et al. Aging patient population with ruptured aneurysms: trend over 28 years. Neurosurgery. 2021;88(3):658-665.
Samuels OB, Sadan O, Feng C, et al. Aneurysmal subarachnoid hemorrhage: trends, outcomes, and predictions from a 15-year perspective of a single neurocritical care unit. Neurosurgery. 2021;88(3):574-583.
Navi BB, Kamel H, McCulloch CE, et al. Accuracy of neurovascular fellows’ prognostication of outcome after subarachnoid hemorrhage. Stroke. 2012;43(3):702-707.
Macdonald RL, Cusimano MD, Etminan N, et al. Subarachnoid hemorrhage international trialists data repository (SAHIT). World Neurosurg. 2013;79(3-4):418-422.
Mascitelli JR, Cole T, Yoon S, et al. External validation of the subarachnoid hemorrhage international trialists (SAHIT) predictive model using the barrow ruptured aneurysm trial (BRAT) cohort. Neurosurgery. 2020;86(1):101-106.
de Jong G, Aquarius R, Sanaan B, et al. Prediction models in aneurysmal subarachnoid hemorrhage: forecasting clinical outcome with artificial intelligence. Neurosurgery. 2021;88(5):427-434.
Woodworth GF, Baird CJ, Garces-Ambrossi G, Tonascia J, Tamargo RJ. Inaccuracy of the administrative database: comparative analysis of two databases for the diagnosis and treatment of intracranial aneurysms. Neurosurgery. 2009;65(2):251-257.
Etminan N, Brown RD, Beseoglu K, et al. The unruptured intracranial aneurysm treatment score: a multidisciplinary consensus. Neurology. 2015;85(10):881-889.
Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373-383.
Fisher CM, Kistler JP, Davis JM. Relation of cerebral vasospasm to subarachnoid hemorrhage visualized by computerized tomographic scanning. Neurosurgery. 1980;6(1):1-9.
Frontera JA, Claassen J, Schmidt JM, et al. Prediction of symptomatic vasospasm after subarachnoid hemorrhage: the modified Fisher scale. Neurosurgery. 2006;59(1):21-27.
Hunt WE, Hess RM. Surgical risk as related to time of intervention in the repair of intracranial aneurysms. J Neurosurg. 1968;28(1):14-20.
Drake CG, Hunt WE, Sano K. Report of World federation of neurological Surgeons committee on a universal subarachnoid hemorrhage grading scale. J Neurosurg. 1988;68(6):985-986.
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-845.
Wan KR, Maszczyk T, See AAQ, Dauwels J, King NKK. A review on microelectrode recording selection of features for machine learning in deep brain stimulation surgery for Parkinson’s disease. Clin Neurophysiol. 2019;130(1):145-154.
Khan O, Badhiwala JH, Grasso G, Fehlings MG. Use of machine learning and artificial intelligence to drive personalized medicine approaches for spine care. World Neurosurg. 2020;140:512-518.
Sarkiss CA, Germano IM. Machine learning in neuro-oncology: can data analysis from 5,346 patients change decision making paradigms? World Neurosurg. Published online ahead of print January 23, 2019. doi: 10.1016/j.wneu.2019.01.046.
doi: 10.1016/j.wneu.2019.01.046
Saber H, Somai M, Rajah GB, Scalzo F, Liebeskind DS. Predictive analytics and machine learning in stroke and neurovascular medicine. Neurol Res. 2019;41(8):681-690.
Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet. 2019;393(10181):1577-1579.
Moons KG, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or disagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-W73.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BJOG. 2015;122(3):434-443.
Hostettler IC, Pavlou M, Ambler G, et al. Assessment of the subarachnoid hemorrhage international trialists (SAHIT) models for dichotomized long-term functional outcome prediction after aneurysmal subarachnoid hemorrhage in a United Kingdom multicenter cohort study. Neurosurgery. 2020;87(6):1269-1276.
Paliwal N, Jaiswal P, Tutino VM, et al. Outcome prediction of intracranial aneurysm treatment by flow diverters using machine learning. Neurosurg Focus. 2018;45(5):E7.
Shiraz Bhurwani MM, Waqas M, Podgorsak AR, et al. Feasibility study for use of angiographic parametric imaging and deep neural networks for intracranial aneurysm occlusion prediction. J Neurointerv Surg. 2020;12(7):714-719.
Guédon A, Thépenier C, Shotar E, et al. Predictive score for complete occlusion of intracranial aneurysms treated by flow-diverter stents using machine learning. J Neurointerv Surg. 2021;13(4):341-346.
Staartjes VE, Sebök M, Blum PG, et al. Development of machine learning-based preoperative predictive analytics for unruptured intracranial aneurysm surgery: a pilot study. Acta Neurochir (Wien). 2020;162(11):2759-2765.
Megjhani M, Terilli K, Frey HP, et al. Incorporating high-frequency physiologic data using computational dictionary learning improves prediction of delasyed cerebral ischemia compared to existing methods. Front Neurol. 2018;9:122.
Tanioka S, Ishida F, Nakano F, et al. Machine learning analysis of matricellular proteins and clinical variables for early prediction of delayed cerebral ischemia after aneurysmal subarachnoid hemorrhage. Mol Neurobiol. 2019;56(10):7128-7135.
Ramos LA, van der Steen WE, Sales Barros R, et al. Machine learning improves prediction of delayed cerebral ischemia in patients with subarachnoid hemorrhage. J Neurointerv Surg. 2019;11(5):497-502.
Capoglu S, Savarraj JP, Sheth SA, Choi HA, Giancardo L. Representation learning of 3D brain angiograms, an application for cerebral vasospasm prediction. Annu Int Conf IEEE Eng Med Biol Soc. 2019;2019:3394-3398.
Megjhani M, Terilli K, Weiss M, et al. Dynamic detection of delayed cerebral ischemia: a study in 3 centers. Stroke. 2021;52(4):1370-1379.
Savarraj JPJ, Hergenroeder GW, Zhu L, et al. Machine learning to predict delayed cerebral ischemia and outcomes in subarachnoid hemorrhage. Neurology. 2021;96(4):e553-e562.
Lo BW, Macdonald RL, Baker A, Levine MA. Clinical outcome prediction in aneurysmal subarachnoid hemorrhage using Bayesian neural networks with fuzzy logic inferences. Comput Math Methods Med. 2013;2013:904860.
Zafar SF, Postma EN, Biswal S, et al. Electronic health data predict outcomes after aneurysmal subarachnoid hemorrhage. Neurocrit Care. 2018;28(2):184-193.
Rubbert C, Patil KR, Beseoglu K, et al. Prediction of outcome after aneurysmal subarachnoid haemorrhage using data from patient admission. Eur Radiol. 2018;28(12):4949-4958.
Tabaie A, Nemati S, Allen JW, et al. Assessing contribution of higher order clinical risk factors to prediction of outcome in aneurysmal subarachnoid hemorrhage patients. AMIA Annu Symp Proc. 2019;2019:848-856.
Xia N, Chen J, Zhan C, et al. Prediction of clinical outcome at discharge after rupture of anterior communicating artery aneurysm using the random forest technique. Front Neurol. 2020;11:538052.
Liu J, Xiong Y, Zhong M, et al. Predicting long-term outcomes after poor-grade Aneurysmal subarachnoid hemorrhage using decision tree modeling. Neurosurgery. 2020;87(3):523-529.
Hostettler IC, Muroi C, Richter JK, et al. Decision tree analysis in subarachnoid hemorrhage: prediction of outcome parameters during the course of aneurysmal subarachnoid hemorrhage using decision tree analysis. J Neurosurg. 2018;129(6):1499-1510.
de Toledo P, Rios PM, Ledezma A, Sanchis A, Alen JF, Lagares A. Predicting the outcome of patients with subarachnoid hemorrhage using machine learning techniques. IEEE Trans Inf Technol Biomed. 2009;13(5):794-801.
Katsuki M, Kakizawa Y, Nishikawa A, Yamamoto Y, Uchiyama T. Easily created prediction model using deep learning software (Prediction One, Sony Network Communications Inc.) for subarachnoid hemorrhage outcomes from small dataset at admission. Surg Neurol Int. 2020;11:374.
Yu D, Williams GW, Aguilar D, et al. Machine learning prediction of the adverse outcome for nontraumatic subarachnoid hemorrhage patients. Ann Clin Transl Neurol. 2020;7(11):2178-2185.
Dengler NF, Madai VI, Unteroberdörster M, et al. Outcome prediction in aneurysmal subarachnoid hemorrhage: a comparison of machine learning methods and established clinico-radiological scores. Neurosurg Rev. 2021;44(5):2837-2846.
Maldaner N, Zeitlberger AM, Sosnova M, et al. Development of a complication- and treatment-aware prediction model for favorable functional outcome in aneurysmal subarachnoid hemorrhage based on machine learning. Neurosurgery. 2021;88(2):E150-E157.
Muscas G, Matteuzzi T, Becattini E, et al. Development of machine learning models to prognosticate chronic shunt-dependent hydrocephalus after aneurysmal subarachnoid hemorrhage. Acta Neurochir (Wien). 2020;162(12):3093-3105.
Wang W, Kiik M, Peek N, et al. A systematic review of machine learning models for predicting outcomes of stroke with structured data. PLoS One. 2020;15(6):e0234722.
Ramspek CL, Jager KJ, Dekker FW, Zoccali C, van Diepen M. External validation of prognostic models: what, why, how, when and where? Clin Kidney J. 2020;14(1):49-58.
Spetzler RF, McDougall CG, Zabramski JM, et al. The barrow ruptured aneurysm trial: 6-year results. J Neurosurg. 2015;123(3):609-617.
Molyneux A, Kerr R, Stratton I, et al. International subarachnoid aneurysm trial (ISAT) of neurosurgical clipping versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: a randomised trial. Lancet. 2002;360(9342):1267-1274.