Evaluation of the accuracy of ChatGPT in answering asthma-related questions

Comprehensive Summary

This study assessed the accuracy and reproducibility of ChatGPT-4’s responses to seven asthma-related questions derived from GINA (Global Initiative for Asthma) guidelines. Two authors independently posed the questions, resetting conversation history each time to avoid bias, and each was asked twice. Six experts scored the answers on a 4-point Likert scale (1 = totally correct, 4 = totally incorrect), evaluating from both specialist and layperson perspectives. Results showed that 85% of ChatGPT’s answers were consistent, with none rated “totally incorrect.” Specialists most often rated responses as “2” (correct but insufficient) or “3” (mixed accuracy), while layperson-focused evaluations more often rated answers as “1” (totally correct). Overall, ChatGPT provided adequate general information but struggled with complex mechanisms, reaffirming that AI should supplement - not replace - professional medical advice.

Outcomes and Implications

Although ChatGPT is accessible and practical, it remains too new to reliably support expert-level clinical decision-making. Still, with 80% of patients open to using ChatGPT for self-diagnosis and many physicians already adopting AI tools, improving its accuracy is clinically urgent. If carefully integrated, ChatGPT could reduce unnecessary visits, lower patient costs, and allow physicians to redirect time toward research or maintaining work–life balance. The work highlights both the immediate limitations and the long-term potential of AI to streamline healthcare delivery.