VALSE论文速览 第185期：Generalized fMRI-to-Image Reconstruction by GESS

论文题目：

Alleviating the Semantic Gap for  Generalized fMRI-to-Image Reconstruction

作者列表：

方涛 (浙江大学)，郑乾 (浙江大学)，潘纲 (浙江大学)

B站观看网址：

https://www.bilibili.com/video/BV1um42157PG/

论文摘要：

Although existing fMRI-to-image  reconstruction methods could predict high-quality images, they do not  explicitly consider the semantic gap between training and testing data,  resulting in reconstruction with unstable and uncertain semantics. This paper  addresses the problem of generalized fMRI-to-image reconstruction by  explicitly alleviates the semantic gap. Specifically, we leverage the  pre-trained CLIP model to map the training data to a compact feature  representation, which essentially extends the sparse semantics of training  data to dense ones, thus alleviating the semantic gap of the instances nearby  known concepts (i.e., inside the training super-classes). Inspired by the  robust low-level representation in fMRI data, which could help alleviate the  semantic gap for instances that far from the known concepts (i.e., outside  the training super-classes), we leverage structural information as a general  cue to guide image reconstruction. Further, we quantify the semantic  uncertainty based on probability density estimation and achieve Generalized  fMRI-to-image reconstruction by adaptively integrating Expanded Semantics and  Structural information (GESS) within a diffusion process. Experimental  results demonstrate that the proposed GESS model outperforms state-of-the-art  methods, and we propose a generalized scenario split strategy to evaluate the  advantage of GESS in closing the semantic gap.

参考文献：

[1] Tao Fang, Qian Zheng, Gang Pan. “Alleviating the Semantic Gap for Generalized fMRI-to-Image Reconstruction”, Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS). 2023.

论文链接：

[https://openreview.net/pdf?id=qSS9izTOpo]

代码链接：

[https://github.com/duolala1/GESS]

视频讲者简介：

方涛，浙江大学博士生，从属潘纲教授团队，方向为脑机接口，研究工作包括基于扩散模型的功能核磁共振成像信号到视觉信号解码，以及基于动作电位的运动信号解码等，相关工作发表在 NeurIPS、AAAI 等国际会议上。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：傅雪阳 (中国科学技术大学)