VALSE Webinar 25-18期 总第389期 医疗大模型的术与道

报告嘉宾：谢伟迪 (上海交通大学)

报告题目：Building Agentic System for Clinical Diagnosis and Evaluation

报告嘉宾：戈宗元 (Monash University)

报告题目：Modern Medical AI with Large-Scale Model

Panel议题：

1. 回顾医疗大模型的发展历史，当前面临的挑战和未来的机遇有哪些？

2. 学术界与产业界在推动医疗大模型发展时的差异在哪些方面？如何打破两者之间的壁垒，实现学术与商业共赢？

3. 医疗大模型如何在实际应用中更好的赋能医疗行业？

4. 医学数据在隐私保护上的限制如何影响医疗大模型的发展？如何推动更广泛的数据共享和跨界合作？

Panel嘉宾：

谢伟迪 (上海交通大学)、戈宗元 (Monash University)、吴贤 (腾讯天衍实验室)、王烁 (复旦大学)、王连生 (厦门大学)

报告嘉宾：谢伟迪 (上海交通大学)

报告时间：2025年6月25日 (星期三)晚上20:05 (北京时间)

报告题目：Building Agentic System for Clinical Diagnosis and Evaluation

报告人简介：

谢伟迪，上海交通大学长聘轨副教授，首批教育部U40获得者，国家级青年人才 (海外)，科技部科技创新 2030 —“新一代人工智能”重大项目青年项目负责人，上海市海外高层次人才计划获得者，上海市启明星计划获得者，国家基金委面上项目负责人。 博士毕业于牛津大学视觉几何组 (Visual Geometry Group，VGG)，首批 Google-DeepMind 全额奖学金获得者，China-Oxford Scholarship获得者，牛津大学工程系杰出奖获得者。主要研究领域为计算机视觉，医学人工智能，共发表论文超 80篇，包括Nature Communications，NPJ Digital Medicine, CVPR，ICCV, NeurIPS, ICML, IJCV等，Google Scholar累计引用 15000次，多次获得国际顶级会议研讨会的最佳论文奖和最佳海报奖，最佳期刊论文奖，MICCAI Young Scientist Publication Impact Award Finalist；Nature Medicine，Nature Communications特邀审稿人，计算机视觉和人工智能领域的旗舰会议CVPR，NeurIPS，ECCV的领域主席。

个人主页：

https://weidixie.github.io

报告摘要：

近年来，基于大模型的医疗人工智能 (AI)取得了显著进展，正在重塑疾病诊断的方式。本次报告将围绕三个方向展开： (1)总体概述语言大模型在疾病诊断领域的最新研究进展、存在的挑战以及未来的发展潜力； (2)探讨如何构建具备检索 (retrieval)，推理 (reasoning)与定位能力 (grounding)的多模态大模型，以提升诊断的准确性与可解释性； (3)分享一个面向罕见病诊断的多智能体系统设计方案，强调其透明性与可溯源性。通过这些内容，报告将展现大模型驱动下医疗AI的最新趋势与实际应用前景。

参考文献：

[1] RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2025. https://arxiv.org/abs/2503.04653

[2] Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases. In: Nature Communications (under revision), https://arxiv.org/abs/2503.04691

[3] Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach. https://arxiv.org/abs/2506.03238

[4] PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents.

In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2023. (MICCAI Young Scientist Publication Impact Award, Finalist)

[5] Knowledge-enhanced Pre-training for Auto-diagnosis of Chest Radiology Images. In: Nature Communications, 2023. https://www.nature.com/articles/s41467-023-40260-7

[6] Towards Building Multilingual Language Model for Medicine. In: Nature Communications, 2024. https://www.nature.com/articles/s41467-024-52417-z

[7] Large-scale Long-tailed Disease Diagnosis on Radiology Images. In: Nature Communications, 2024. https://www.nature.com/articles/s41467-024-54424-6

[8] Towards Evaluating and Building Versatile Large Language Models for Medicine. In: Npj Digital Medicine (Nature Portfolio), 2025. https://www.nature.com/articles/s41746-024-01390-4

[9] Development of a large-scale medical visual question-answering dataset (PMC-VQA). In: Nature Communications Medicine, 2025. https://www.nature.com/articles/s43856-024-00709-2

[10] Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D & 3D Medical Data. In: Nature Communications (in press), 2025. https://arxiv.org/abs/2308.02463

[11] A Knowledge-enhanced Pathology Vision-language Foundation Model for Cancer Diagnosis. Nature Cancer (under revision), 2025. https://arxiv.org/abs/2412.13126

[12] One Model to Rule them All: Towards Universal Segmentation for Medical Images with Text Prompts. In: Npj Digital Medicine (2nd review). https://arxiv.org/abs/2312.17183

报告嘉宾：戈宗元 (Monash University)

报告时间：2025年6月25日 (星期三)晚上20:35 (北京时间)

报告题目：Modern Medical AI with Large-Scale Model

报告人简介：

Associate Professor Zongyuan Ge conducts interdisciplinary research at the boundary between Medical Artificial Intelligence, Computer-aided Diagnosis, Biomedical Engineering, Digital Health, Medical Imaging and Machine Learning and is a multi-award-winning medical information science and technology entrepreneur. His research leverages cutting-edge AI technologies using large-scale multi-modality medical data including imaging, biological signal, medical records, genomics data, multi-omics and models the clinicians’ medical knowledge underlying tasks like diagnosis, prognosis, disease management and treatment for eye (ophthalmology), skin (dermatology), heart (cardiovascular) and neurodegeneration diseases such as epilepsy, dementia and multiple sclerosis. He is also one of Australia’s most in-demand experts in advanced technology, including medical robotics, digital health and artificial intelligence, and is a passionate science communicator. His standing as a top 2% highly cited researcher since 2020 and recognition of his articles in Clarivate Analytics' list of highly cited work reaffirm the impact of his research, and he is recognised as the 200 Most Qualified Young Researchers in Computer and Mathematics by the Scientific Committee of the Heidelberg Laureate Foundation (2017).

个人主页：

https://zongyuange.github.io/

报告摘要：

This talk will define and motivate the problem of Medical Al Research, the challenges as well as recent progress at the Monash AIM for Health Lab. This includes component technologies such as deep learning, evaluation methodologies, human-ai interaction, and also end-to-end software-to-hardware systems for applications such as disease diagnosis, disease prediction and management, and treatment.

The aim is to develop next-generation medical applications capabilities using the LLM and Foundation Model that reduce Australia's increasingly burden of disease, and that enable transformation of Australia's most important healthcare sectors through enhanced automation and medical Al technologies. Our research re-combines expertise in human stem cell biology, biomedical engineering, clinical research and artificial intelligence scientists. Our medical Al products have served millions of people in China, India, Japan, South Africa and Australia.

This talk will highlight the transformative impact of foundation models in medical artificial intelligence, showcasing their application across diverse clinical domains. We will specifically explore our pioneering work in developing the Comprehensive Al Retinal Expert (CARE) system, a foundation model published in The Lancet Digital Health, which leverages over 230,000 color fundus photographs and is scaling with 100 million images for robust retinal analysis. Further-more, we will introduce PanDerm, a novel foundation model for dermatology presented in Nature Medicine, which integrates four complementary imaging techniques to enhance early-stage melanoma detection and improve diagnostic accuracy for both specialists and non-specialists.

These advancements, alongside our efforts in neurology Al for epilepsy management, underscore the potential of foundation models to revolutionize medical diagnosis, treatment, and ultimately, patient outcomes by effectively harnessing large-scale, multi-modality medical data.

参考文献：

[1] Yan, S., Yu, Z., Primiero, C., Vico-Alonso, C., Wang, Z., Yang, L., ... & Ge, Z. (2025). A General-Purpose Multimodal Foundation Model for Dermatology. Nature Medicine.

[2] Gong, S., Zhang, X., Nguyen, X.A., Shi, Q., Lin, F., Chauhan, S., Ge, Z. and Cheng, W., 2023. Hierarchically resistive skins as specific and multimetric on-throat wearable biosensors. Nature Nanotechnology, pp.1-9.

[3] Cheng, X., Zhong, Y., Harandi, M., Dai, Y., Chang, X., Li, H., Drummond, T. and Ge, Z., 2020. Hierarchical neural architecture search for deep stereo matching. Advances in Neural Information Processing Systems, 33, pp.22158-22169.

[4] Bewley, A., Ge, Z., Ott, L., Ramos, F. and Upcroft, B., 2016, September. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP) (pp. 3464-3468). IEEE.

[5] Hakeem, H., Feng, W., Chen, Z., Choong, J., Brodie, M. J., Fong, S. L., ... & Ge, Z., Kwan, P. (2022). Development and validation of a deep learning model for predicting treatment response in patients with newly diagnosed epilepsy. JAMA neurology, 79(10), 986-996

[6] Ju, L., Wang, X., Wang, L., Mahapatra, D., Zhao, X., Zhou, Q., Liu, T. and Ge, Z., 2022. Improving Medical Images Classification with Label Noise Using Dual-Uncertainty Estimation. IEEE transactions on medical imaging.

[7] Yu, Z., Nguyen, J., Nguyen, T.D., Kelly, J., Mclean, C., Bonnington, P., Zhang, L., Mar, V. and Ge, Z., 2021. Early Melanoma Diagnosis with Sequential Dermoscopic Images. IEEE Transactions on Medical Imaging, 41(3), pp.633-646.

[8] Lin, D., Xiong, J., Liu, C., Zhao, L., Li, Z., Yu, S., Wu, X., Ge, Z., Hu, X., Wang, B. and Fu, M., 2021. Application of Comprehensive Artificial intelligence Retinal Expert (CARE) system: a national real-world evidence study. The Lancet Digital Health, 3(8), pp.e486-e495.

[9] Hu, M., Wang, L., Yan, S., Ma, D., Ren, Q., Xia, P., ... & Ge, Z. (2023). Nurvid: A large expert-level video database for nursing procedure activity understanding. Advances in Neural Information Processing Systems, 36, 18146-18164.

[10] Yan S, Yu Z, Zhang X, Mahapatra D, Chandra SS, Janda M, Soyer P, Ge Z. Towards trustable skin cancer diagnosis via rewriting model's decision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 11568-11577)

Panel嘉宾：吴贤 (腾讯天衍实验室)

嘉宾简介：

吴贤博士是腾讯天衍实验室主任，专家研究员。吴博士的主要研究兴趣包括自然语言理解，深度机器学习，医学大模型等。吴博士在Nature Biomedical Engineering, Nature Computational Science, npj digital medicine，T-PAMI, NeurIPS, ACL, CVPR等国际顶级杂志会议上发表文章一百余篇，被引用超过6000次，有近50项美国和中国专利。吴博士获得过华夏医学科技一等奖，在国际MSCOCO评测中获得过第一名，也在ICDM知识图谱评测中获得了第二名的成绩。在加入腾讯之前，吴博士先后供职于IBM研究院和微软人工智能和研究部门。吴博士在上海交通大学获得计算机博士学位。

个人主页：

https://scholar.google.com/citations?hl=zh-TW&user=lslB5jkAAAAJ

Panel嘉宾：王烁 (复旦大学)

嘉宾简介：

王烁，复旦大学数字医学研究中心青年研究员，博士生导师，兼任英国伦敦帝国理工学院荣誉资深研究员，中国计算机学会数字医学分会副秘书长，上海市力学学会生物力学专委会委员，上海市医学会数字医学分会委员，上海市内镜微创协同创新中心首席研究员，医学图像学术会议MICCAI领域主席，BioMedical Engineering Online 等医工杂志副主编。主要研究领域为多模态医学人工智能，领导复旦大学数字医学研究中心AIM3 (AI for Multi-Modal Medicine)课题组，致力于发展面向专病诊疗的医学人工智能基础模型和临床转化，已在医工领域发表文章 90 余篇，包括 Nature Biomedical Engineering、Nature Medicine、Nature Machine Intelligence、Nature Communications 等期刊和CVPR、AAAI等会议，并发起CMRx系列国际医学影像挑战赛。

个人主页：

https://swang.miccai.cloud/

Panel嘉宾：王连生 (厦门大学)

嘉宾简介：

王连生，现为厦门大学信息学院教授，医学院双聘教授，博士生导师，数字福建健康医疗大数据研究所副所长，福建省医学会放射学分会AI学组副组长，海医会智能医学影像与信息化专委会副主任，厦门大学医学院医学人工智能研究院负责人，MICS主席。长期从事医学影像处理研究，主持和参与多项科研项目，包括国家自然科学基金仪器专项、科技部科技创新2030重大项目、国家重点研发项目、国家自然科学基金面上和青年项目等，发表包括Nature Machine Intelligence、Nature Communications、Cell Reports Methods、Cell Patterns、人工智能顶会CVPR/AAAI等相关研究论文100余篇，获得腾讯犀牛鸟科研奖、CSPE Young Investigator、福建省科技进步二等奖、2023年厦门大学田昭武交叉学科一等奖，带领团队先后11次在国际医学影像比赛中获得冠军。

个人主页：

https://xmu-lswang.github.io/

主持人：陈浩 (香港科技大学)

主持人简介：

陈浩，香港科技大学计算机科学与工程系，化学与生物工程系和生命科学部助理教授，医工交叉联合创新中心主任，研究兴趣包括大模型医疗，计算病理，多模态数据融合，医学图像分析，计算机辅助微无创诊疗等。在Nature Biomedical Engineering、Nature  Communications、Lancet Digital Health、Nature Machine Intelligence、MICCAI、IEEE-TMI、MIA、CVPR、ICCV等顶级期刊和会议发表论文100余篇 (谷歌学术引用34000余次，h-index 76)，连续入选斯坦福大学全球排名前2%科学家名单，科睿唯安全球高被引科学家等。曾获得2023年亚洲青年科学家、国家教育部优秀成果二等奖、北京市科技进步一等奖、2019年人工智能医学影像顶级会议MICCAI青年科学家影响力奖等奖项，担任包括IEEE TMI、TNNLS、J-BHI等期刊编委，担任ICLR、CVPR、ACM MM、MICCAI等多个国际会议的领域主席和程序委员，曾带领团队获得15项国际医学图像分析的挑战赛冠军。

个人主页：

https://smartlab.cse.ust.hk/

特别鸣谢本次Webinar主要组织者：

主办AC：陈浩 (香港科技大学)