A systematic comparison of ChatGPT and DeepSeek for guideline-based question answering in obstetric anesthesia

Comprehensive Summary

With recent advances in artificial intelligence (AI), the applicability of Large Language Models (LLMs) in the field of obstetric anesthesia is being considered. The current study performs a systematic comparison between four LLMs: ChatGPT-4o, ChatGPT-4o-mini, DeepSeek-V3, and DeepSeek-R1 in an effort to evaluate the models’ ability to answer guideline-based clinical questions on obstetric anesthesia. The questions were asked with the following prompting strategies: Isolated Prompting (IP), Batch Prompting (BP), and Contextual Isolated Prompting (CIP). The responses generated by the LLMs were assessed for accuracy, overconclusiveness, supplementary value, and completeness. The results of the study find no significant differences in the dimensions of accuracy (p = 0.743), overconclusiveness (p = 0.118), and completeness (p = 0.391), but find significant differences in supplementary values (p = 0.008) with ChatGPT models performing better than DeepSeek. Using the Flesh Reading Ease scores, it was determined that ChatGPT-4o demonstrated the highest readability out of the LLM models. With regard to prompting strategies, CIP produced responses that provided more clinically relevant responses.

Outcomes and Implications

The similarity in the performance of the four LLMs across various dimensions suggests a uniform, comparable level of clinical reasoning ability in contemporary models. ChatGPT models prompted with CIP provided clinically rich responses seemingly outperforming DeepSeek models, which influences which model may be considered for use in the future of obstetric anesthesia. The differences in response quality dependent on prompting style suggests a variability in the LLMs that would need to be addressed before the models can be implemented as clinical support tools in obstetric anesthesia.