Comprehensive Summary
In this study, a machine learning algorithm was developed and validated for identifying patients with neuroinfectious diseases (NID) using unstructured clinical notes. From Mass General Brigham (January 22, 2010 to September 21, 2023), 3000 notes from patients who had undergone lumbar puncture were evaluated by 6 NID-expert physicians to establish ground truth. To create a natural language processing-based (NLP) model, an extreme gradient boosting (XGBoost) framework was used to classify NID cases, and performance was evaluated using the area under the receiver operating curve (AUROC) and the precision-recall curve (AUPRC). Of the clinical note language analyzed, the most significant words were “meningitis,” “ventriculitis,” and “meningoencephalitis.” The NLP model achieved an AUROC of 0.98 (95% CI 0.96-0.99) and AUPRC of 0.89 (95% CI 0.83-0.94) on the MGB test data. The NLP model outperformed NID identification assessed through International Classification of Diseases billing codes and via Llama 3.2 on both accuracy and reliability on MGB data. This model was further validated after being tested on external data from Beth Israel Deaconess Medical Center (BIDMC).
Outcomes and Implications
Unlike previous research that only used observable symptoms, structured data, or had limited validation, this NLP-based model was validated in 2 independent datasets that classifies patients using unstructured EHR notes. In settings where structured lab data is unavailable, this can potentially serve as a vital tool for identifying NID in a cohort. Further model development will build off this study’s limitations and expand training to diverse sources across the United States and potentially include microbial data or antimicrobial therapy as a data point.