Comprehensive Summary
The study done by Xu et.al, looks at how large language model perplexity (a measure of the level of "surprise" of an LLM after reading a sentence based on context) can be used to measure semantic coherence in speech. Semantic coherence is an important indicator of thought disorganization in schizophrenia spectrum disorders (SSDs). The purpose of this study was to determine if LLM-derived metrics can complement traditional semantic proximity-based methods (methods in which an LLM predicts subsequent words based on words in proximity to them) to improve automatic detection of incoherent speech. The study used two datasets, the AVH dataset and the Clinical Interview dataset. The AVH dataset consisted of audio diaries from 202 participants with auditory verbal hallucinations and the Clinical Interview consisted of transcripts from 39 schizophrenia outpatients with clinician-rated disorganization scores. The authors used both "chain" and "bag" methods for determining perplexity, representing cumulative context and central static context respectively. The results found that the chain model performed the best among the perplexity models and the best performance was achieved by combining perplexity and proximity features, not by using either alone.
Outcomes and Implications
The authors found that LLMs measuring speech incoherence using perplexity and proximity methods complement each other. The proximity metrics captured the semantic similarity the best and the perplexity metrics captured continuity and predictability. It is beneficial for these methods to work together in order to properly reflect the two complementary aspects of coherence which is semantic memory and working memory. These findings have important implications for the detection of risk for SSDs. This approach combining perplexity and proximity shows promise for automated diagnosis and monitoring of schizophrenia-spectrum disorders, advancing the use of AI to detect linguistic biomarkers for mental health.