Comprehensive Summary
This study, conducted by Hones et al., evaluates ChatGPT’s response regarding wrist arthritis surgery. The researchers gathered a list of 32 questions related to common surgery techniques and presented them to ChatGPT three times. They reviewed the responses and rated 75% of all the responses and all three responses for 23 out of 32 questions as appropriate. Moreover, an intraclass correlation coefficient (ICC) was calculated to be 0.97 which indicates that the answers to the same questions have excellent reliability. Despite giving mostly acceptable answers, ChatGPT’s accuracy varied across procedure, requiring caution and skepticism when accessing information with it.
Outcomes and Implications
ChatGPT is a Large Language Model which is trained to generate human-like responses. With doctors and patients having easy access to its tool, it is wildly used to answer any question, including those related to wrist arthritis. It is essential to evaluate the consistency of the AI model’s answers, as previous studies only reported generally good quality answers from ChatGPT. This study confirms the potential for inaccuracies in LLM’s answers and the need to be used in conjunction with information from experts. Nevertheless, studies like these need frequent update, as this study used ChatGPT 3.5 instead of the most recent version.