VALSE 论文速览 第95期：面向跨模态匹配的噪声关联学习

论文题目：Learning with Noisy Correspondence for Cross-modal Matching

作者列表：黄振宇 (四川大学)、牛国成 (百度)、刘霄 (好未来)、丁文彪 (好未来)、肖欣延 (百度)、吴华 (百度)、*彭玺 (四川大学)

B站观看网址：

https://www.bilibili.com/video/BV1ne41137vY/

论文摘要：

Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge number of cross-modal matching methods have been proposed and achieved remarkable progress in recent years, almost all of these methods implicitly assume that the multimodal training data are correctly aligned. In practice, however, such an assumption is extremely expensive even impossible to satisfy. Based on this observation, we reveal and study a latent and challenging direction in cross-modal matching, named noisy correspondence, which could be regarded as a new paradigm of noisy labels. Different from the traditional noisy labels which mainly refer to the errors in category labels, our noisy correspondence refers to the mismatch paired samples. To solve this new problem, we propose a novel method for learning with noisy correspondence, named Noisy Correspondence Rectifier (NCR). In brief, NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via an adaptive prediction model in a co-teaching manner. To verify the effectiveness of our method, we conduct experiments by using the image-text matching as a showcase. Extensive experiments on Flickr30K, MS-COCO, and Conceptual Captions verify the effectiveness of our method.

论文信息：

[1] Huang, Z., Niu, G., Liu, X., Ding, W., Xiao, X., Wu, H., & Peng, X. (2021). Learning with Noisy Correspondence for Cross-modal Matching. Advances in Neural Information Processing Systems, 34, 29406-29419.

论文链接：

[https://proceedings.neurips.cc/paper/2021/file/f5e62af885293cf4d511ceef31e61c80-Paper.pdf]

[https://proceedings.neurips.cc/paper/2021/file/f5e62af885293cf4d511ceef31e61c80-Supplemental.pdf]

代码链接：

[https://github.com/XLearning-SCU/2021-NeurIPS-NCR]

视频讲者简介：

黄振宇，四川大学博士生，师从彭玺教授。主要研究方向包括多模态学习、噪声标签学习和深度聚类等。在TIP、NeurIPS、ICML、CVPR和IJCAI等CCF A类期刊和会议发表多篇论文。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：冯尊磊 (浙江大学)，徐易 (大连理工大学)

季度责任AC：张姗姗 (南京理工大学)