Comprehensive Summary
This study evaluates the accuracy, consistency, and intermodel reliability of four large language models (LLMs), ChatGPT-4o (OpenAI), Google Gemini, Perplexity AI, and Microsoft Copilot, using official Fellowship of the Royal Colleges of Surgeons (FRCS) Section 1 single-best-answer (SBA) questions across ten surgical specialties. Each model was tested on 50 questions presented in triplicate to account for stochastic variability, generating 600 total responses. Results demonstrated significant performance differences. ChatGPT achieved the highest overall accuracy (81.3%, p<0.0001), excelling in cardiothoracic surgery and neurosurgery (100% accuracy), while Gemini underperformed in neurosurgery (40%) and urology (20%). All models showed weaker performance in otolaryngology and plastic surgery. Consistency rates were high (84–90%), but intermodal agreement was low, highlighting variability in model reasoning. Within-model reliability was strongest for Perplexity and Gemini, with ChatGPT coming in last as the least stable out of the three. While ChatGPT outperformed peers in accuracy, Gemini and Perplexity were more consistent across repeated trials. Findings highlight the potential of LLMs as assistant study tools for surgical trainees but underscore limitations in specialty-specific dependability. Ethical concerns include over-reliance and innate dataset bias, especially regarding underrepresented populations. Future directions include specialty-specific fine-tuning and rigorous validation before incorporating the LLMs into surgical education and clinical decisions.
Outcomes and Implications
The implications of this study suggest that while large language models (LLMs) like ChatGPT show promise as supportive tools in surgical education, they are not yet reliable enough to be used independently in high-stakes contexts. For the medical community, these models may serve as useful secretaries or assistants for exam preparation and knowledge reinforcement, but their variability across specialties and inconsistency in repeated responses highlight the need for further refinement. This is especially true for when it comes to specialty-specific training, with strict validation vital to implement before clinical integration. For the public, the findings underscore both the potential and the limitations of AI in healthcare, as such tools could help train surgeons and expand access to medical knowledge worldwide. However, their current shortcomings mean they must be used cautiously, with safeguards against reliance and bias to ensure fairness, accuracy, and patient safety. In conclusion, the takeaway is that while AIs like ChatGPT can be very helpful study aids for future surgeons, they are not yet reliable enough to replace human learning or decision-making. There are also risks: relying too much on AI could cause mistakes, and because these systems are mostly trained on Western data, they may miss important differences in patients from other backgrounds. More research, larger studies, and better-designed medical AIs are needed before these tools can be safely used in real-life surgical training or patient care.