Impact of artificial intelligence in managing musculoskeletal pathologies in physiatry: a qualitative observational study evaluating the potential use of ChatGPT versus Copilot for patient information and clinical advice on low back pain

Comprehensive Summary

In this work, Ah-Yan et al. investigated the potential of Large Language Models (LLMs) to support patients in understanding and managing musculoskeletal disorders. Specifically, the authors compared the ability of ChatGPT (OpenAI) and Copilot (Microsoft) to provide information and advice about low back pain. With input from 18 physiatrists and 3 physiatry residents, the authors developed a list of 27 questions covering different aspects of low back pain. These questions were submitted to both LLMs with and without a hypothetical patient’s clinical context. Three independent physiatrists evaluated the responses for validity, safety, and usefulness using a Likert scale from 1 (least favorable) to 4 (most favorable). The evaluations were consistent, and no clinically significant differences were found between the models. Both ChatGPT and Copilot scored above 3 in all categories, and the inclusion of clinical context did not significantly affect their performance. These results suggest that both models can assist patients in self-managing low back pain. However, this study has limitations: it relied on static data, unblinded evaluators, and subjective interpretation of the rating scale. Moreover, there was no comparison between the model’s responses and those provided by physicians. These factors should be considered before recommending LLMs for direct patient support.

Outcomes and Implications

Low back pain is a leading cause of disability, with self-management through patient information offering significant benefits. This study revealed that both ChatGPT and Copilot can provide valid, safe, and useful medical advice related to low back pain, regardless of whether a clinical context is provided. Therefore, these models could support patients in managing their condition independently. Despite current availability, several concerns regarding the use of AI-generated medical advice remain. Notably, ensuring transparency and accountability is critical to mitigate the possible risks and harms. Future studies are needed to evaluate the safety of LLMs over time in this context.