VALSE Webinar 24-27期 总第362期 医学视觉语言大模型：进展与展望 ...

报告嘉宾：郑冶枫 (西湖大学)

报告题目：Medical Imaging Meets Vision-Language Model

报告嘉宾：李响 (Massachusetts General Hospital and Harvard Medical School)

报告嘉宾：章恺 (Lehigh University)

报告题目：BiomedGPT: A generalist vision–language foundation model for diverse biomedical tasks

报告嘉宾：周洪宇 (Harvard Medical School)

报告题目：Learning to diagnose whispers of the human body

报告嘉宾：郑冶枫 (西湖大学)

报告时间：2024年9月18日 (星期三)晚上20:00 (北京时间)

报告题目：Medical Imaging Meets Vision-Language Model

报告人简介：

郑冶枫, 本科和硕士毕业于清华大学电子工程系，博士毕业于美国马里兰大学电气与计算机工程系。2006年至2017年在西门子医疗美国研究院工作；2018年至2024年担任腾讯杰出科学家和天衍实验室主任；2024年7月全职加入西湖大学。他的研究方向是医学人工智能，包括医学影像智能分析、自然语言处理、医学多模态大模型。他已经发表论文300多篇，拥有美国发明专利90多项，论文被引用22,000多次，h-index指数74。他是国际电气和电子工程师协会会士（IEEE Fellow）、美国医学和生物工程学会会士（AIMBE Fellow），现担任IEEE医学影像杂志副编，曾经担任医学影像AI的顶会MICCAI 2021大会程序委员会主席和多个人工智能顶会的领域主席 (包括NeurIPS, AAAI, IJCAI和MICCAI)。

个人主页：

https://www.westlake.edu.cn/faculty/yefeng-zheng.html

报告摘要：

近年来大语言模型技术取得飞速的进步，也被逐步应用于包括医疗在内的各个垂直领域。在实际临床中，医生要结合不同的信息来源（不同的模态），综合判断，做疾病的诊断，并为病人制定个性化的治疗方案。医学影像和电子病历是最常见的两个模态。本次分享将介绍我们在医学视觉文本大模型方面的一些初步探索，包括融合医学先验知识的基座模型预训练，影像报告的自动生成，文本辅助的病理影像诊断，最后将介绍我们最近提出的医学大模型自动评估技术。

参考文献：

[1] Wang et al., “An Inclusive Task-Aware Framework for Radiology Report Generation,” Proceedings of Medical Imaging Computing and Computer Aided Intervention, 2022.

[2] Liu et al., “Improving Medical Vision-Language Contrastive Pretraining with Semantics-aware Triage,” IEEE Transactions on Medical Imaging, 2023.

[3] Liu et al., “A Multimodal Large Language Modelling Deep Learning Framework for the Future Pandemic,” npj Digital Medicine, 2023.

[4] Shi et al., "ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification,” Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

[5] Wang et al., “imapScore: Medical Fact Evaluation Made Easy,” Findings of Annual Conf. Association for Computational Linguistics, 2024

报告嘉宾：李响 (哈佛大学医学院和麻省总医院)

报告时间：2024年9月18日 (星期三)晚上20:30 (北京时间)

报告人简介：

李响教授是哈佛大学医学院和麻省总医院放射科的助理教授。他领导了多个医学影像、文本分析和多模态融合项目，专注于开发医疗健康领域的人工智能（AI）解决方案。李响教授的研究主要聚焦在人工通用智能（AGI）在医学数据的开发和应用上，以应对在复杂临床环境中的实际挑战，包括多机构数据的异质性、模型的可扩展性和计算限制，以及临床工作流程的系统集成。他在医学影像和文本分析方法、人工智能进行疾病诊断和检测，生成式人工智能，以及医疗大数据的计算架构设计等主题上于Nature Medicine，TPAMI，ICML，ACL等期刊和会议上发表了130多篇文章，h-index指数38。

个人主页：

https://researchers.mgh.harvard.edu/profile/15451263/

报告嘉宾：章恺 (Lehigh University)

报告时间：2024年9月18日 (星期三)晚上20:30 (北京时间)

报告题目：BiomedGPT: A generalist vision–language foundation model for diverse biomedical tasks

报告人简介：

Kai Zhang is a Ph.D. student in the Department of Computer Science & Engineering at Lehigh University, where he is advised by Prof. Lichao Sun. Kai has extensive experience in research and development of AI for real-world applications. He has interned at SRA, Amazon, and NEC, where he developed prototypes for edge computing, fraud detection, and radiology interpretation. Additionally, he was a research intern at the AI&I Department, Mayo Clinic, where he worked on vision-language pre-training and federated learning algorithms for biomedicine. Kai has published over 10 papers in the AI domain, with his recent research focusing on trustworthy medical AI, particularly in enhancing explainability and reliability to facilitate practical deployment.

个人主页：

https://taokz.github.io/

报告摘要：

Traditional biomedical artificial intelligence (AI) models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners and patients. In this talk, We will discuss the development and performance of BiomedGPT, a novel open-source, lightweight vision-language foundation model designed as a generalist AI for biomedical applications. Unlike previous solutions, BiomedGPT is both computationally efficient and accessible, achieving state-of-the-art results in 16 out of 25 benchmarks across a variety of tasks. We will present human evaluation results that underscore its effectiveness in radiology visual question answering, report generation, and summarization, with performance metrics nearing human expert levels. This talk will explore how BiomedGPT exemplifies the potential of a multi-modal, generalist approach to revolutionize medical diagnostics and improve workflow efficiency.

参考文献：

[1] Zhang, K., Zhou, R., Adhikarla, E. et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine (2024). https://doi.org/10.1038/s41591-024-03185-2

报告嘉宾：周洪宇 (Harvard Medical School)

报告时间：2024年9月18日 (星期三)晚上21:00 (北京时间)

报告题目：Learning to diagnose whispers of the human body

报告人简介：

Dr. Hong-Yu Zhou is a postdoctoral fellow at Harvard Medical School’s Department of Biomedical Informatics, where he focuses on integrating artificial intelligence with medicine and data science. His research explores scalable solutions for medical AI, emphasizing multimodal analytics and large-scale model design. Dr. Zhou completed his PhD at The University of Hong Kong, where he was advised by Prof. Yizhou Yu, and holds prior degrees from Nanjing University and Wuhan University. His work has been featured in top-tier conferences and journals, such as Nature Biomedical Engineering, Nature Machine Intelligence, TPAMI, ICLR, and MICCAI. He has also received several prestigious awards for his contributions to medical imaging and AI.

个人主页：

https://zhouhy.org/

报告摘要：

In this talk, I will discuss my research on developing artificial intelligence systems for computational precision health, emphasizing multimodal learning and integration techniques that help interpret diverse patient data. This talk will cover three fundamental challenges: enhancing medical decision accuracy through learning multimodal representations, harnessing large-scale, high-quality, yet low-cost health insights, and developing generalist medical AI with specialized capabilities. I will also highlight the real-world impact of these innovations, showcasing improvements in the diagnostics and prognostics of lung diseases, breast cancer treatment response prediction, and increased efficiency in radiologists’ report writing. This presentation will explore the transformative potential of AI in improving medical decision making and its role in advancing personalized medicine.

主持人：于乐全 (香港大学)

主持人简介：

Dr. Lequan Yu is an Assistant Professor at the Department of Statistics and Actuarial Science, The University of Hong Kong. Before joining HKU, he was a postdoctoral research fellow at Stanford University. He obtained his Ph.D. degree from The Chinese University of Hong Kong in 2019 and bachelor’s degree from Zhejiang University in 2015. His research interests are developing advanced machine learning methods for biomedical data analysis, with a primary focus on medical images. He has been named on the World's First List of Top 150 Chinese Young Scholars in Artificial Intelligence and ranked by Clarivate Analytics in the top 1% of the citation list. He has also won the MICCAI 2023 Young Scientist Publication Impact Award Runner-Up, CUHK Young Scholars Thesis Award, and Best Paper Award of Medical Image Analysis-MICCAI in 2017. He serves as the area chair/senior PC member of MICCAI, IJCAI, AAAI, and the regular reviewer for top-tier journals and conferences.

个人主页：

https://yulequan.github.io/

特别鸣谢本次Webinar主要组织者：

主办AC：于乐全 (香港大学)