Large Language Models in Neurological Practice: Real-World Study

Comprehensive Summary

This study by Maiorana et al. evaluates the ability of freely available AI chatbots, such as ChatGPT and Google Gemini, to diagnose neurological conditions compared to human neurologists. The researchers tested these AI models using 28 real patient cases from a neurology hospital in Italy, and presenting the same clinical information that neurologists had at patient admission and comparing their diagnostic accuracy and test recommendations. Neurologists correctly diagnosed 75% of cases, outperforming ChatGPT (54% accuracy) and Gemini (46% accuracy). Both AI models struggled with complex cases requiring nuanced clinical judgment and frequently overprescribed diagnostic tests in 17-25% of cases, with ChatGPT requiring additional prompting in about one-third of cases to provide complete responses. The authors concluded that while AI models show promise as supportive tools, they currently lack the depth needed for independent clinical decision-making without specialized training, and their human-like conversational style may create risks of misunderstanding in clinical settings.

Outcomes and Implications

This research is important because it reveals the limitations of using consumer-grade AI chatbots as diagnostic aids in real-world clinical practice, where physicians may turn to these tools during routine patient care without structured protocols. The study demonstrates that freely available AI models perform adequately for straightforward conditions like vascular disorders but fall short in complex cases involving movement disorders, psychiatric presentations, or rare autoimmune conditions, precisely where expert clinical judgment matters most. The tendency of these AI systems to recommend excessive diagnostic testing could lead to unnecessary procedures and increased healthcare costs. While the authors suggest AI could eventually support neurologists when properly integrated with human oversight, they emphasize that current consumer-grade models require significant refinement and specialized training before safe clinical implementation, and future medical professionals will need education on appropriate AI use to avoid over-reliance on automated suggestions.