Comprehensive Summary
The authors explore the need for a standardized test set when validating Artificial Intelligence (AI) models. Two AI pediatric wrist fracture detection models, EfficientNet (for fracture classification) and YOLOv11 (for fracture localization), were trained and validated on 18,762 radiographs. The models were then evaluated on two equally sized test sets of 4,588 images: one constructed using a balanced sampling strategy incorporating case difficulty, projection type, and fracture presence, and the other generated through random selection. Performance metrics significantly decreased in the balanced test set with more challenging cases; the precision and Precision-Recall (PR) area under the curve (AUC) for YOLOv11 models decreased from 0.95 and 0.911 in the random set to 0.83 and 0.732 in the balanced set. In addition, the precision and AUC for EfficiantNet models decreased from 0.779 ± 0.011 and 0.940-0.899 in the random set to 0.760 ± 0.019 and 0.870-0.769 in the balanced set. It is important to note that this study used a single institution pediatric wrist radiograph dataset, limiting generalizability to other body regions, age groups, and institutions.
Outcomes and Implications
Discrepancies within test sets used to validate medically related AI algorithms could lead to confusing and misleading results. This may cause a model to be clinically implemented when it is far from ready for use, patients to be incorrectly diagnosed, surgeries to be botched, and a range of other problems; this obviously depends on what the model is and how it is implemented. Although the author does not explicitly address clinical implementation, future studies are suggested to further validate the results. In addition, these findings should be considered by all researchers currently engaging in medically related AI research.