VALSE

VALSE 首页 活动通知 查看内容

VALSE 论文速览 第101期:基于动态梯度调节的平衡化多模态学习 ...

2022-11-18 17:17| 发布者: 程一-计算所| 查看: 858| 评论: 0

摘要: 为了使得视觉与学习领域相关从业者快速及时地了解领域的最新发展动态和前沿技术进展,VALSE最新推出了《论文速览》栏目,将在每周发布一至两篇顶会顶刊论文的录制视频,对单个前沿工作进行细致讲解。本期VALSE论文速 ...

为了使得视觉与学习领域相关从业者快速及时地了解领域的最新发展动态和前沿技术进展,VALSE最新推出了《论文速览》栏目,将在每周发布一至两篇顶会顶刊论文的录制视频,对单个前沿工作进行细致讲解。本期VALSE论文速览选取了来自中国人民大学的多模态学习方面的工作。该工作由胡迪助理教授指导、论文共同第一作者卫雅珂博士生录制。


论文题目:Balanced Multimodal Learning via On-the-fly Gradient Modulation

作者列表:彭小康 (中国人民大学)、卫雅珂 (中国人民大学)、邓安东 (上海交通大学)、王栋 (上海人工智能实验室)、胡迪 (中国人民大学)

B站观看网址:

https://www.bilibili.com/video/BV1mW4y177SF/


论文摘要:

Multimodal learning helps to comprehensively under- stand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility.


论文信息:

[1] X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu. Balanced multimodal learning via on-the-fly gradient modulation. In CVPR, 2022.


论文链接:

[https://arxiv.org/pdf/2203.15332.pdf]


代码链接:

[https://github.com/GeWu-Lab/OGM-GE_CVPR2022]


视频讲者简介:

卫雅珂,中国人民大学高瓴人工智能学院博士生。主要研究方向为多模态学习。在TPAMI、CVPR等顶级期刊和会议发表多篇论文。



特别鸣谢本次论文速览主要组织者:

月度轮值AC:李冠彬 (中山大学)、李镇 (香港中文大学 (深圳))

季度责任AC:张姗姗 (南京理工大学)


活动参与方式

1、VALSE每周举行的Webinar活动依托B站直播平台进行,欢迎在B站搜索VALSE_Webinar关注我们!

直播地址:

https://live.bilibili.com/22300737;

历史视频观看地址:

https://space.bilibili.com/562085182/ 


2、VALSE Webinar活动通常每周三晚上20:00进行,但偶尔会因为讲者时区问题略有调整,为方便您参加活动,请关注VALSE微信公众号:valse_wechat 或加入VALSE QQ R群,群号:137634472);


*注:申请加入VALSE QQ群时需验证姓名、单位和身份缺一不可。入群后,请实名,姓名身份单位。身份:学校及科研单位人员T;企业研发I;博士D;硕士M。


3、VALSE微信公众号一般会在每周四发布下一周Webinar报告的通知。


4您也可以通过访问VALSE主页:http://valser.org/ 直接查看Webinar活动信息。Webinar报告的PPT(经讲者允许后),会在VALSE官网每期报告通知的最下方更新。

小黑屋|手机版|Archiver|Vision And Learning SEminar

GMT+8, 2024-3-29 15:24 , Processed in 0.013443 second(s), 14 queries .

Powered by Discuz! X3.4

Copyright © 2001-2020, Tencent Cloud.

返回顶部