精准肿瘤学的视觉语言基础模型

Clinical decision-making is driven by multimodal data, including clinical notes and pathological characteristics. Artificial intelligence approaches that can effectively integrate multimodal data hold significant promise in advancing clinical care1,2. However, the scarcity of well-annotated multimodal datasets in clinical settings has hindered the development of useful models. In this study, we developed the Multimodal transformer with Unified maSKed modeling (MUSK), a vision-language foundation model designed to leverage large-scale, unlabelled, unpaired image and text data. MUSK was pretrained on 50 million pathology images from 11,577 patients and one billion pathology-related text tokens using unified masked modelling. It was further pretrained on one million pathology image-text pairs to efficiently align the vision and language features. With minimal or no further training, MUSK was tested in a wide range of applications and demonstrated superior performance across 23 patch-level and slide-level benchmarks, including image-to-text and text-to-image retrieval, visual question answering, image classification and molecular biomarker prediction. Furthermore, MUSK showed strong performance in outcome prediction, including melanoma relapse prediction, pan-cancer prognosis prediction and immunotherapy response prediction in lung and gastro-oesophageal cancers. MUSK effectively combined complementary information from pathology images and clinical reports and could potentially improve diagnosis and precision in cancer therapy.

临床决策是由多模态数据驱动的，包括临床记录和病理特征。能够有效整合多模态数据的人工智能方法在推进临床护理方面具有重大前景。然而，临床环境中注释良好的多模态数据集的稀缺阻碍了有用模型的开发。在这项研究中，我们开发了具有统一建模（MUSK）的多模态变换器，这是一种视觉语言基础模型，旨在利用大规模、未标记、未配对的图像和文本数据。使用统一的掩码建模对来自11577名患者的5000万张病理图像和10亿个病理相关文本标记进行了MUSK预训练。对100万对病理图像文本对进行了进一步的预训练，以有效地对齐视觉和语言特征。在很少或没有进一步培训的情况下，MUSK在广泛的应用中进行了测试，并在23个补丁级和幻灯片级基准测试中表现出卓越的性能，包括图像到文本和文本到图像检索、视觉问答、图像分类和分子生物标志物预测。此外，MUSK在结果预测方面表现强劲，包括肺癌和食道癌的黑色素瘤复发预测、全癌预后预测和免疫疗法反应预测。MUSK有效地结合了来自病理图像和临床报告的补充信息，并有可能提高癌症治疗的诊断和准确性。