Comprehensive Summary
Santucci et al present a study using specific parameters to evaluate information regarding interstitial cystitis found by patients on 4 public AI models: ChatGPT-4.0, Perplexity, ChatSonic, and Bing AI. A set of ten separate queries related to interstitial cystitis were created using Google Tables and Hopkins Medical website and put into the 4 AI models. The answers were then evaluated by two blind, independent reviewers utilizing specific guidelines such as PEMAT-P and DISCERN. The answers were evaluated for understandability, actionability, and readability. The answers from each chatbot were compared to each other as well. The chatbots showed a median quality with Perplexity scoring the highest and Bing AI scoring the lowest. However, across all the AI models used the overall readability of the information provided was deemed too difficult for the average person to understand. The way the chatbots were prompted also greatly changed the responses provided, such as making them more actionable for patients. Overall the study showed that there was no great amounts of misinformation provided by the chatbots.
Outcomes and Implications
This study has important implications for the medical community because as the use of AI chatbots to learn about medical conditions increase, patients must be given the tools to sift through the information they are given to find what is helpful to them. Much further testing must be done as many of the queries used in this study are from a general point of view rather than more specific and personal questions that patients may ask a chatbot. But, it is the job of healthcare providers to give patients as many resources as they can to learn about their illness and with the rise of AI, it needs to be made sure that patients are given guidelines to follow so that they can have the comfort of knowing they are getting truthful information.