为了使得视觉与学习领域相关从业者快速及时地了解领域的最新发展动态和前沿技术进展,VALSE最新推出了《论文速览》栏目,将在每周发布一至两篇顶会顶刊论文的录制视频,对单个前沿工作进行细致讲解。本期VALSE论文速览选取了来自浙江大学的脑视觉信号解码的工作。该工作由郑乾和潘纲教授指导,论文一作方涛同学录制。 论文题目: Alleviating the Semantic Gap for Generalized fMRI-to-Image Reconstruction 作者列表: 方涛 (浙江大学),郑乾 (浙江大学),潘纲 (浙江大学) B站观看网址: 论文摘要: Although existing fMRI-to-image reconstruction methods could predict high-quality images, they do not explicitly consider the semantic gap between training and testing data, resulting in reconstruction with unstable and uncertain semantics. This paper addresses the problem of generalized fMRI-to-image reconstruction by explicitly alleviates the semantic gap. Specifically, we leverage the pre-trained CLIP model to map the training data to a compact feature representation, which essentially extends the sparse semantics of training data to dense ones, thus alleviating the semantic gap of the instances nearby known concepts (i.e., inside the training super-classes). Inspired by the robust low-level representation in fMRI data, which could help alleviate the semantic gap for instances that far from the known concepts (i.e., outside the training super-classes), we leverage structural information as a general cue to guide image reconstruction. Further, we quantify the semantic uncertainty based on probability density estimation and achieve Generalized fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and Structural information (GESS) within a diffusion process. Experimental results demonstrate that the proposed GESS model outperforms state-of-the-art methods, and we propose a generalized scenario split strategy to evaluate the advantage of GESS in closing the semantic gap. 参考文献: [1] Tao Fang, Qian Zheng, Gang Pan. “Alleviating the Semantic Gap for Generalized fMRI-to-Image Reconstruction”, Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS). 2023. 论文链接: [https://openreview.net/pdf?id=qSS9izTOpo]
代码链接: [https://github.com/duolala1/GESS]
视频讲者简介: 方涛,浙江大学博士生,从属潘纲教授团队,方向为脑机接口,研究工作包括基于扩散模型的功能核磁共振成像信号到视觉信号解码,以及基于动作电位的运动信号解码等,相关工作发表在 NeurIPS、AAAI 等国际会议上。 特别鸣谢本次论文速览主要组织者: 月度轮值AC:傅雪阳 (中国科学技术大学) 活动参与方式 1、VALSE每周举行的Webinar活动依托B站直播平台进行,欢迎在B站搜索VALSE_Webinar关注我们! 直播地址: https://live.bilibili.com/22300737; 历史视频观看地址: https://space.bilibili.com/562085182/ 2、VALSE Webinar活动通常每周三晚上20:00进行,但偶尔会因为讲者时区问题略有调整,为方便您参加活动,请关注VALSE微信公众号:valse_wechat 或加入VALSE QQ T群,群号:863867505); *注:申请加入VALSE QQ群时需验证姓名、单位和身份,缺一不可。入群后,请实名,姓名身份单位。身份:学校及科研单位人员T;企业研发I;博士D;硕士M。 3、VALSE微信公众号一般会在每周四发布下一周Webinar报告的通知。 4、您也可以通过访问VALSE主页:http://valser.org/ 直接查看Webinar活动信息。Webinar报告的PPT(经讲者允许后),会在VALSE官网每期报告通知的最下方更新。 |
小黑屋|手机版|Archiver|Vision And Learning SEminar
GMT+8, 2024-10-31 14:19 , Processed in 0.015162 second(s), 14 queries .
Powered by Discuz! X3.4
Copyright © 2001-2020, Tencent Cloud.