Multimodal AI generates virtual population for tumor microenvironment modeling

Comprehensive Summary

Tumor immune microenvironment is a strong influence in cancer progression and response to immunotherapy. Usage of TIME with multiplex immunofluorescence (mIF) for imaging gives substantial information but tends to be expensive and has a slow processing rate. Valanarasu et al. developed a multimodal deep-learning framework (GigaTIME) that translates routine hematoxylin & eosin (H&E) pathology slides into virtual multiplex immunofluorescence (mIF) images. By utilizing over 40 million individual cells from pathology slides with both H&E and mIF data, GigaTIME was able to learn the morphological features of H&E to correspond to specific protein expression patterns. The model was able to output mIF predictions for 21 markers, including immune and tumor proteins. This was then utilized for 14,256 cancer patients across 51 hospitals and 1000 clinics to produce 299,376 virtual mIF whole-slide images, covering 24 cancer types and 306 subtypes. The researchers identified 1,234 statistically significant associations between GigaTIME–predicted protein activations and clinical attributes such as biomarkers (e.g., tumor mutation burden), pathological staging, and survival outcomes. Moreover, this allowed for patient stratification of the signatures across 21-protein channels and demonstrated spatial patterns and protein interactions occurring; these correlated with clinical biomarkers and outcomes stronger and more accurately than before.

Outcomes and Implications

The GigaTIME model allows for high-resolution and widespread immune profiling by using existing pathology infrastructure at a low cost. In addition, this model allows for possible new biomarker identification, target discovery, and more through the mIF slides. This also allows for better patient stratification, prognosis prediction, and potentially treatment selection.