Performance of Large Language Models in the Japanese Public Health Nurse National Examination: Comparative Cross-Sectional Study

Comprehensive Summary

This study looked at how large language models performed on the 111th Japanese Public Health Nurse National Examination. The researchers tested 3 language learning models: GPT-4o, Claude Opus 4, and Gemini 2.5 Pro. The study used all 110 exam questions and were categorized by format, content, and selection type. Accuracy rates and 95% confidence intervals were calculated and chi-square tests were used for comparisons. All three models scored above the passing standard of 60%. Gemini 2.5 Pro achieved 92.7% accuracy, Claude Opus 4 scored 91.8%, and GPT-4o scored 85.5%. All models showed lower accuracy on multiple choice questions than on single choice questions but overall performed well.

Outcomes and Implications

The results suggest that large language models can help support public health nursing education. Their high accuracy shows they can answer many questions related to community and public health nursing correctly. However, the model performances dropped on multiple choice questions, indicating that they lack the reasoning required for accurate answering and choice differentiation. This suggests that complex reasoning tasks are still challenging for current models. LLMs can serve as helpful learning tools but should be used carefully in specialized nursing education.