Evaluating ChatGPT-4 as a digital patient education tool in anesthesia: A multi-rater quality assessment

Comprehensive Summary

This study by Akçaalan et al. investigated the quality of ChatGPT-4 responses to commonly asked patient questions about anesthesia. Twenty-two common anesthesia-related questions were identified through online search and submitted to ChatGPT-4. These questions included topics like anxiety about receiving anesthesia, potential perioperative risks involved, and implications for women who are breastfeeding or pregnant. The responses to the twenty-two questions were then rated individually by five anesthesiology specialists on a Likert-type scale (1 = excellent response not requiring clarification; 2 = satisfactory requiring minimal clarification; 3 = satisfactory requiring moderate clarification; 4 = unsatisfactory requiring substantial clarification). Across the 110 collected ratings of the responses, 61.8% were considered excellent (1), 32.7% were satisfactory, requiring minimal clarification (2), and 5.5% were satisfactory despite requiring moderate clarification (3). Notably, none of the responses were deemed unsatisfactory (4) by any of the raters. The average scores for individual questions ranged from 1.0 to 2.4, while average scores across individual raters ranged from 1.27 to 1.73. Overall, ChatGPT-4 was able to provide mostly effective answers to commonly asked questions about anesthesia based on the subjective evaluation of five anesthesiology specialists.

Outcomes and Implications

Since newer and improved models of ChatGPT will likely continue to emerge, relatively high current response quality for frequently asked patient questions about anesthesia, as determined by five anesthesiology specialists, indicates its potential as an effective patient education tool. However, given the small cohort of selected raters and the moderate variability in scoring between them, it is important to consider the need for professional oversight when supplementing patient education with large language models like ChatGPT-4, as responses can often be context-dependent. Nonetheless, this study highlights the current effectiveness of ChatGPT-4 in providing basic clinical information regarding anesthesia-related patient questions and a framework for future investigation.