Comprehensive Summary
May et al. evaluated whether Uro_Chat, a newly developed large language model trained on European Association of Urology (EAU) guidelines, could pass the 2022 In-Service Assessment of the European Board of Urology (EBU). The AI model was tested on 100 multiple-choice questions in three separate rounds, each spaced by 48 hours, to assess consistency and accuracy. The model achieved 62%, 62%, and 63% accuracy respectively, meeting the minimum passing score of 61% for the EBU exam. Uro_Chat performed much stronger when prompted on oncologic topics compared with non-oncologic ones. The consistency of results were also fairly high throughout all 3 rounds. While Uro_Chat slightly underperformed compared with the average human candidate (68% mean accuracy), its near-passing scores convey that an AI model trained with domain-specific medical data can approach the knowledge competency of professional examinees.
Outcomes and Implications
This study highlights the potential of domain-focused language models in the clinical field, especially in medical assessment and education. Uro_Chat’s performance compared to real trainees was not significantly different, showing its promise as a supplementary tool. However, its uneven performance between oncologic and non-oncologic topics suggest the need for broader training beyond narrow subspecialties. In the future, similar AI models could serve as useful tools for exam preparation and even clinical usage for patients. Still, physician oversight is required to ensure safety and accuracy in medical training.