VALSE 论文速览 第101期：基于动态梯度调节的平衡化多模态学习 ...

论文题目：Balanced Multimodal Learning via On-the-fly Gradient Modulation

作者列表：彭小康 (中国人民大学)、卫雅珂 (中国人民大学)、邓安东 (上海交通大学)、王栋 (上海人工智能实验室)、胡迪 (中国人民大学)

B站观看网址：

https://www.bilibili.com/video/BV1mW4y177SF/

论文摘要：

Multimodal learning helps to comprehensively under- stand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility.

论文信息：

[1] X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu. Balanced multimodal learning via on-the-fly gradient modulation. In CVPR, 2022.

论文链接：

[https://arxiv.org/pdf/2203.15332.pdf]

代码链接：

[https://github.com/GeWu-Lab/OGM-GE_CVPR2022]

视频讲者简介：

卫雅珂，中国人民大学高瓴人工智能学院博士生。主要研究方向为多模态学习。在TPAMI、CVPR等顶级期刊和会议发表多篇论文。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：李冠彬 (中山大学)、李镇 (香港中文大学 (深圳))

季度责任AC：张姗姗 (南京理工大学)