Comprehensive Summary
This article, published by Mihail et al. looks to see if DeepSeek-R1's reasoning LLM was able to match, or even outperform, OpenAI's o1 model. To conduct this head to head evaluation, 442 cases taken from clinical settings, used to assess diagnostic accuracy between models. The cases were taken from 10 specific ophthalmology subspecialties, with n>50 for retina&vitreous, neuro-ophthalmology, uveitis, and pediatrics. For diagnostic accuracy, the Deepseek model correctly assessed 70.4% cases (297 of 422) compared to OpenAi's score of 63.0% (266 of 422). When assessing the correct management plan, Deepseek also outperformed open AI, with a score of 82.7% compared to 75.8%. Next, both models went through cost analysis to assess economical efficiency, where Deepseek came out to be 1.5% of OpenAI's total cost. With both trials, Deepseek surpassed its AI counterpart with lower operational costs. Some limitations with this study is a limited sample size for specific specialties, as well as difficulties in assessing text.
Outcomes and Implications
The training of LLMs as a support for ophthalmology diagnosis has been progressing in the recent past, and with the introduction of Deepseek-R1, a much cheaper alternative has become viable. Some fields, such as glaucoma, have high diagnostic accuracy, showing that these models are better suited for some tasks as opposed to others. Training is still required for Deepseek to become clinically viable, however it shows a clear path to its development.