This study by Chang et al. evaluates the perfomance of several AI chatbots as clinical psychiatric support tools through the usage of Rasch analysis, a statistical method typically used to quantify item difficulty. The researchers selected 160 multiple choice questions from the Taiwan Psychiatry Licensing Examination, which were then given to 27 chatbots through a zero-shot testing approach. Each chatbot was given the same prompt, and the researchers prompted additional explanations for certain questions with a standardized query in order to further understand the chatbots’ reasoning. Overall, the chatbots averaged 61% accuracy, with a standard deviation of 19.5. The highest performing chatbot, ChatGPT-o1-preview, had an accuracy rate of 80.6%. It generally excelled in diagnostic ability, pharmacological principles, and general therapeutic concepts. However, it was limited in its ability to recall precise factual details and also exhibited biases in its reasoning. The observed limitations of the chatbots highlight the importance of training data quality, as they are particularly susceptible to misinformation which poses a potential risk to patient safety. Even so, chatbots hold significant potential as psychiatric tools to give feedback, as long as they are implemented safely and with human oversight.