External validation of an artificial intelligence tool for fracture detection in children with osteogenesis imperfecta: a multireader study

Comprehensive Summary

This study evaluates whether a commercially available artificial intelligence (AI) tool can support radiologists in identifying fractures on radiographs of children with osteogenesis imperfecta (OI). An AI model developed by Milvue was used, trained on over 600,000 general chest and musculoskeletal radiographs, but not on rare skeletal disorders such as OI. Seven radiologists reviewed 336 radiographs from 48 pediatric OI patients in two reading rounds, first without and then with AI assistance, using consultant radiologist interpretation as ground truth. The model achieved an accuracy of 74.8% compared to 83.4% for radiologists alone. However, combined radiologist-AI interpretation increased accuracy to 90.7%, with intra-reader agreement increasing from 0.52 to 0.74 per examination. Further, if the radiologists did not catch a fracture that the model did, the radiologists would change their decision, leading to a correct result 82.8% of the time, with 64% of these corrections addressing false positives. Despite inferior standalone performance, AI assistance improved diagnostic accuracy by reducing radiologist false positives, highlighting the value of collaboration over autonomous AI in rare pediatric disorders.

Outcomes and Implications

OI is a difficult disease to diagnose on radiographs due to distorted bone shapes and the presence of artifacts that may shield visibility, such as previous fractures. Concern for missed fractures may drive overcalling, leading to unnecessary bone casts and emotional stress for children and their families. AI can act as a second reader, prompting physicians to re-evaluate suspected fractures and confirm the model’s findings. While there are concerns of automation bias, or over-reliance on AI without proper judgement, this study demonstrated that physicians can potentially implement the Milvue model as a valuable tool instead of depending wholly on it to improve overall fracture detection outcomes. Further development is necessary to train AI on more equitable data that accounts for rare conditions as well as general ones, validate on international OI datasets, and compare with other relevant models. However, this commercially available tool demonstrates potential as a supportive aid to leverage AI pattern recognition alongside clinician judgement to improve fracture detection in children with OI.