VALSE 论文速览 第90期：面向动态视音场景的问答学习机制

论文题目：Learning to Answer Questions in Dynamic Audio-Visual Scenarios

作者列表：李光耀 (中国人民大学)，卫雅珂 (中国人民大学)，田亚鹏 (University of Rochester)，徐辰良 (University of Rochester)，文继荣 (中国人民大学)，胡迪 (中国人民大学)

B站观看网址：

https://www.bilibili.com/video/BV1SB4y1k7xZ/

论文摘要：

In this paper, we focus on the Audio-Visual Question Answering (AVQA)task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos. The problem requires comprehensive multimodal understanding and spatio-temporal reasoning over audio-visual scenes. To benchmark this task and facilitate our study, we introduce a large-scale MUSIC-AVQA dataset, which contains more than 45K question-answer pairs covering 33 different question templates spanning over different modalities and question types. We develop several baselines and introduce a spatio-temporal grounded audio-visual network for the AVQA problem. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-, V-, and AVQA approaches. We believe that our built dataset has the potential to serve as testbed for evaluating and promoting progress in audio-visual scene understanding and spatio-temporal reasoning.

论文信息：

[1] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, Di Hu, Learning to Answer Questions in Dynamic Audio-Visual Scenarios, CVPR 2022 (Oral presentation)

论文链接：

[https://openaccess.thecvf.com/content/CVPR2022/papers/Li_Learning_To_Answer_Questions_in_Dynamic_Audio-Visual_Scenarios_CVPR_2022_paper.pdf]

代码链接：

[https://github.com/GeWu-Lab/MUSIC-AVQA]

项目主页链接：

[https://gewu-lab.github.io/MUSIC-AVQA/]

知乎讲解：

[https://zhuanlan.zhihu.com/p/498943693]

视频讲者简介：

李光耀，中国人民大学高瓴人工智能学院博士研究生，主要研究方向是多模态视频理解。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：王智慧 (大连理工大学)、杨旭 (西安电子科技大学)

季度责任AC：魏秀参 (南京理工大学)