Comprehensive Summary
The study examined if ChatGPT-3.5 was capable of providing accurate and comprehensive information to common questions regarding living kidney donation (LKD). The authors selected 22 most frequently asked questions about LKD then asked five bilingual nephrologists and six bilingual donors/donor candidates to rate the AI-generated responses in English and French using a 5-point Likert scale. For English responses, Nephrologists rated 50% of the answers as accurate, 59% as comprehensive, and 27% as clear. However, donors/candidates gave much higher ratings in all three categories, especially for accuracy (91%) and comprehensiveness (100%). Both the nephrologists and donors/candidates rated the French responses much lower, ranging from 45–9% for nephrologists and 0-77% among donors/candidates. It is also notable that readability scores were very low, as the responses were written at a college graduate level. Overall, ChatGPT-3.5’s responses demonstrated inconsistency and limited suitability for patient education.
Outcomes and Implications
The study highlights the inability of ChatGPT-3.5 as an educational resource in transplant medicine. While the donors perceived many English responses as accurate and comprehensive, there were significant differences between what the donors saw as clear compared to those of experts. Such discrepancies depict that the ChatGPT is not a reliable tool for medical advice. In addition, the low readability score and low ratings for French responses both underline the limited accessibility to the information to the broader public. While the high ratings on English responses show the possibility of AI serving as a supplementary resource once accuracy and readability are improved, as of now ChatGPT should not be used as a main source for education or advice in a clinical context.