Shifts in emergency physicians' attitudes toward large language model-based documentation: a pre- and post-implementation study

Comprehensive Summary

Lee et al. conducted a prospective longitudinal survey study examining the impact of a large language model (LLM) assistant on emergency department (ED) discharge notes documentation and provider attitudes toward artificial intelligence integration in clinical practice. In November 2024, the Y-KNOT-EDN (Your-Knowledgeable Navigator of Treatment-Emergency department Discharge Note assistant), an in-hospital LLM-based clinical documentation system, was deployed at Severance Hospital in Seoul, South Korea. The system automatically generates a draft discharge note from the patient’s electronic health record (EHR), which physicians then review, modify, and cosign to maintain full clinician oversight. Validated surveys were administered to eight emergency attending physicians at three time points: before implementation (T1), three days post-implementation (T2), and five weeks post-implementation (T3). On average, participants authored 10.9 LLM-assisted discharge notes during the study period. Four provider concerns: loss of control, worsening of patient care, generation of impersonal drafts, and legal and ethical issues, declined significantly over time (p = 0.002, 0.004, 0.010, and 0.028, respectively). Four additional concerns: generation of false information, privacy, impairment of physician reasoning, and data bias, also trended downward, though without reaching statistical significance (p = 0.089, 0.143, 0.156, and 0.268, respectively). Perceived documentation workload declined significantly by 37% from T1 to T3 (p = 0.040). Analysis of documentation time revealed a marked reduction in documentation time, from a mean of 127.5 seconds for manual notes at T1, to 42.8 seconds for LLM-assisted notes at T3 (p = 0.002). Qualitative responses to open-ended questions indicated broad agreement that the LLM assistant improved efficiency by offloading nonclinical administrative documentation tasks. While participants recognized generally high accuracy with fewer errors than anticipated, they identified notable gaps including insufficient patient-specific context for complex cases, absent explanations for abnormal test results, and omission of imaging and laboratory findings. Blinded physician review of the LLM-assisted discharge notes demonstrated high performance across five evaluated domains: fluency, coherence, relevance, safety, and consistency. Mean domain scores on a five-point scale (1 = lowest, 5 = highest) were 4.55, 4.60, 4.72, 4.73, and 4.78, respectively, indicating consistently strong performance.

Outcomes and Implications

This study is among the first to prospectively capture shifting provider perceptions toward AI integration in clinical documentation. This pre- and post-implementation design demonstrated a 26% reduction in provider concerns and a 37% decrease in perceived workload, with provider acceptance remaining consistently high throughout. Key limitations include a small sample size (n = 8), a single-center design that restricts generalizability, and the absence of objective EHR log data for the pre-implementation phase. This forced a reliance on recall-based methods for time estimates, preventing a direct, objective comparison between manual and LLM-assisted documentation speeds.