Melan-Dx: a knowledge-enhanced vision-language framework improves differential diagnosis of melanocytic neoplasm pathology

Comprehensive Summary

This article presents a recently developed AI framework for diagnosis of melanocytic neoplasms, called “Melan-Dx.” The framework integrates textual information from the Penn MD Knowledge Atlas, paired with its visual features to diagnose abnormalities similarly to clinicians. Yao’s design utilizes a dual arm technology, with a “knowledge arm” and “visual arm” working simultaneously to formulate case-wise diagnoses from a medical perspective. The visual arm collects visual data of the abnormality while the knowledge arm consults the medical atlas on the particular features of the abnormality. Rather than matching images within a database, the model analyzes each case individually.

Outcomes and Implications

While dermatopathologists diagnose abnormalities through matching visual analysis with extensive background knowledge and experience, most AI models utilize paired image-text training rather than medical knowledge. This makes those models more susceptible to misdiagnosis and low sensitivity. Through integration of medical knowledge, AI models can progress towards the sensitivity and reasoning of human dermatopathologists. The author notes that such frameworks show potential for growth in pathology foundation models.