Comprehensive Summary
This study evaluates whether GPTs, specifically Llama 2, can extract seizure outcomes from unstructured epilepsy clinic notes without manual annotation. Using over 78,000 notes from the University of Pennsylvania Health System, researchers compared the Llama 2 GPT with prompt engineering against Bio + ClinicalBERT, a fine-tuned LLM trained on biomedical literature and electronic health records. The best Llama 2 prompt achieved 74% accuracy, outperforming Bio + ClinicalBERT without fine-tuning (25%) but falling short of fine-tuned Bio + ClinicalBERT (84%). Notably, in notes written by general neurologists, Llama 2 (76%) surpassed fine-tuned Bio + ClinicalBERT (67%).
Outcomes and Implications
This study shows that GPTs could help clinicians and researchers make better use of unstructured EHR notes without the need for labor-intensive manual annotation. By reliably identifying whether patients with epilepsy are seizure-free, GPTs could enable faster tracking of outcomes, more efficient identification of treatment responses, and earlier recognition of health disparities. Importantly, the models generalized the notes of general neurologists better than epilepsy specialists, suggesting that GPTs could be particularly useful for generalists. At the bedside, this could translate into more timely awareness of seizure recurrence and support for treatment adjustments. However, risks remain as GPTs showed signs of sex-based bias and may misclassify outcomes in certain groups. Further validation in larger and more diverse datasets will be essential to ensure reliability and equity of these models.