Moving beyond word error rate to evaluate automatic speech recognition in clinical samples: Lessons from research into schizophrenia-spectrum disorders

Comprehensive Summary

This study examines the performance of automatic speech recognition in transcribing speech from individuals with schizophrenia spectrum disorders and highlights the limitations of relying solely on word error rate to assess accuracy. Using OpenAI’s Whisper model, the authors found moderate error rates that varied with factors such as symptom severity and led to measurable differences between automated and manual transcripts. These differences affected downstream natural language processing measures and weakened associations with clinical symptom scores. The findings emphasize that evaluating ASR for mental health applications should account for the type and clinical significance of errors, especially in sensitive or high-risk clinical contexts.

Outcomes and Implications

This study has important implications for the medical field by highlighting both the potential and the limitations of automatic speech recognition in clinical practice. While ASR can enable scalable, efficient analysis of patient speech and reduce the burden of manual transcription, the findings show that transcription errors can meaningfully affect clinical interpretations and AI driven decision tools. The results underscore the need for careful evaluation of ASR systems, attention to bias and symptom related error patterns, and the inclusion of human oversight when ASR is used in sensitive settings. Overall, the study supports cautious and context aware integration of speech based AI technologies into medical care