Comprehensive Summary
The study dives into the usage of ChatGPT-3.5, an artificial intelligence chatbot, in hematologic cancers. The researchers compare the performance of the chatbot using questions differentiated to be either general therapy questions or drug-specific. To conduct the study, the researchers first posed ten questions related to hematologic cancers to ChatGPT 3.5: five questions regarding general patient diagnosis, and five specific therapy questions. The average score reflected by ChatGPT for general diagnosis questions was 3.38 on a scale from 1-5, showing that the responses provided by the chatbot were accurate to a limited extent. However, when it came to the drug-specific questions, the average score was 3.06. The scores represent the knowledge bank of the chatbot, demonstrated by its poor performance on questions related to recent developments in the field of hematologic cancers. A limitation to the chatbot considered by the researchers was the fact that the AI model's knowledge bank was limited to 2021 and earlier, which could be a potential cause of its inaccuracy in the questions related to newer developments in the field. Through the study's findings, ChatGPT can be effective regarding general pieces of information and concepts, but less reliable when it comes to specific or recent treatment information.
Outcomes and Implications
The research shows information provided by AI could be outdated or inaccurate, which is why it is important to patients, since they often seek medical information regarding treatment options and drug information using AI. In a clinical setting, AI chatbots such as ChatGPT could potentially improve patients' knowledge of their disease in a general sense. However, the information they gather cannot be equivalent to that which is given by a medical expert, especially when it comes to decisions regarding treatment. In order to implement the usage of AI chatbots in a clinical setting, improvements regarding accuracy and recent developments are needed. The researchers suggest that newer versions of the AI chatbot need to be analyzed to determine if the study's findings are inaccurate.