Comprehensive Summary
This study discussed how physicians’ experience can affect AI-assisted detection of unruptured intracranial aneurysms (UIAs) on 3D TOF MRA. Four models were tuned on an in-house group (between 2019 and 2021) and cross-checked on a public dataset. The two best models (nnDetection and an object-detection “Assis” model) have similarities with five physicians (two non-experts, one moderate, and two experts). Standalone AI has an 86.8% sensitivity with 0.58 false positives per case. Using AI, the expert physicians were able to achieve 88.6% sensitivity and 0.076 false positives per case. The nonexpert physicians had a 72.1% sensitivity and a 0.037 false positive rate, and the moderately knowledgeable physicians had a 78.5% sensitivity with 0.037 false positives per case. This showed that there was some agreement among the more experienced groups and that the misses trended to smaller aneurysms. The FP/TP locations and patterns were recognized differently by each group, with the non-expert physicians rejecting more correct AI flags at the MCA divisions, while the experts added more detections than AI. The authors of the study concluded that performance differs from individual metrics and varies depending on experience.
Outcomes and Implications
The implications of this study are similar to problems with other AI detection technologies. Clinical outcomes depend on how physicians use AI and models have to be evaluated as to not overstate or oversimplify real-world cases. The findings of this study indicate that external validation and guidance to physicians should be done before the technology is used widely, and that, for non-experts, uncertain AI-flagged candidates should be accepted more so that the risk of missing UIAs is much lower. In addition, training should emphasize some of the more complicated regions, like the MCA branch and small-lesion recognition. The authors also discuss that the technology should be implemented with a dataset that has a mix of cases and a design that speaks to the experience of physicians without negating ones with less experience.