AI still not ready for clinical use against COVID-19

Despite its big potential in boosting healthcare quality, artificial intelligence (AI) still seems immature to be clinically deployed against the novel coronavirus disease (COVID-19) pandemic, according to a recent Singapore systematic review.

“Despite widespread interest in novel technologies for the COVID-19 pandemic, our systematic review of the literature reveals that current AI applications were limited in both the range of applications and clinical applicability,” the researchers said. “Several significant issues in the development, validation, and reporting of AI applications undermine safe and effective implementation of these systems within intensive care units or emergency departments.”

Searching through the databases of PubMed, Embase, CINAHL, ACM Digital Library, IEEE Xplore, and Scopus yielded 14 studies eligible for review. Eleven papers looked at predictive AI models: eight developed prognostic models and three focused on diagnosis. The remaining three studies either used existing AI models or considered other outcomes. [Int J Environ Res Public Health 2021;18:4749]

Of the three studies that looked at AI as a diagnostic tool, two developed models that tried to predict COVID-19 status at admission to the emergency department; only one was externally validated. Both models included leukocyte or subpopulation counts as a predictor. The other diagnostic study used AI in a decision-tree to determine COVID-19 status in intensive care based on plasma inflammatory markers.

Across all three cases, performance was decent to very good, with accuracies ranging from 60.5 percent to 98 percent. Nevertheless, assessments for accuracy and subsequent validation differed from study to study, disallowing direct comparison.

For prognosis, AI was mostly used for predicting the need for mechanical ventilation, intensive care admission, or a composite of severe or critical illness. For these types of application, C-indices ranged from 0.79–0.98. One study deployed AI has a prognostic model for in-hospital mortality and found a C-index value of 0.901.

Aside from diagnosis and prognosis, one paper applied AI to optimize resource use, using an artificial immune system algorithm to determine the best queueing model for hospital bed allocation in the intensive care unit. The final model determined an optimal admission rate and number of beds, balanced against costs associated with increasing capacity and turning patients away. Estimates obtained, however, were not validated.

“Our study is the first systematic review of AI applications for COVID-19 in intensive care and emergency settings,” the researchers said. “Applications were largely limited to diagnostic and prognostic predictive modeling, with only one study investigating a separate application of simulating ICU bed occupancy for resource optimization.”

“Due to high risk of bias, inadequate validation, or poor adherence to reporting standards in all reviewed studies, we have found no AI application for COVID-19 ready for clinical deployment in intensive care or emergency settings,” they added.

In turn, the researchers recommended that AI-specific reporting guidelines be integrated in future studies and publications, as well as foster greater collaboration between disciplines.