VALSE Webinar 20220608-14期 总第281期 特邀报告 如何更好地理解对比学习 ... ...

特邀报告嘉宾：田渊栋 (Meta AI Research (FAIR))

报告题目：Towards Better Understanding of Contrastive Learning

*欢迎大家在下方留言提出主题相关问题，主持人会从中选择若干热度高的问题请报告嘉宾进行回复！

特邀报告嘉宾：田渊栋 (Meta AI Research (FAIR))

报告时间：2022年06月08日 (星期三)上午10:00 (北京时间)

报告题目：Towards Better Understanding of Contrastive Learning

报告人简介：

田渊栋博士，Meta/ Facebook人工智能研究院研究员及高级经理，研究方向为深度强化学习，表示学习和优化。曾获得2021年国际机器学习大会 (ICML) 杰出论文奖提名 (Outstanding Paper Honorable Mentions)，及2013年国际计算机视觉大会 (ICCV) 马尔奖提名 (Marr Prize Honorable Mentions)，多次担任NeurIPS，AAAI，AIStats领域主席。围棋开源项目ELF OpenGo项目中研究及工程负责人和第一作者。2013-2014年在Google无人驾驶团队任软件工程师。2005年及08年于上海交通大学获本硕学位，2013年于美国卡耐基梅隆大学机器人研究所获博士学位。

个人主页：

https://yuandong-tian.com/

报告摘要：

While self-supervised learning (SSL) has achieved impressive empirical success, how it works still remains an open problem, in particular when it couples with deep nonlinear network. The talk covers our recent findings on understanding contrastive learning, a popular learning paradigm in SSL. We try to explain the reason of dimensional collapsing in CL, showing how the training dynamics affects the learned representation. We find that a broad family of contrastive learning objective, including InfoNCE loss, is equivalent to a coordinate-wise optimization problem in which the additional min player puts more emphasis on difficult sample pairs with similar representations, in addition to the regular max player that finds good representation to maximize contrastiveness. The resulting formulation, called αCL, unifies not only various existing contrastive losses, which differ by how sample-pair importance α is constructed, but also leads to novel contrastive losses that show comparable (or better) performance on CIFAR10 and STL-10 than classic InfoNCE. In addition, under the same framework, we also discover that in one and two-layer setting, nonlinearity leads to many local optima, each capturing one pattern in the data, while linear activation can only capture the dominant pattern and is not able to learn diverse features. These findings suggest that big models with lots of parameters can be regarded as a brute-force way to find these local optima induced by nonlinearity, a possible underlying mechanism why empirical observations such as the lottery ticket hypothesis hold.

参考文献：

[1] Jing, Li, Pascal Vincent, Yann LeCun, and Yuandong Tian. "Understanding dimensional collapse in contrastive self-supervised learning." ICLR2022

[2] Tian, Yuandong. "Deep Contrastive Learning is Provably (almost) Principal Component Analysis." arXiv preprint arXiv:2201.12680 (2022).

主持人：刘日升 (大连理工大学)

主持人简介：

刘日升，大连理工大学几何计算与智能媒体技术研究所教授，博导，所长。大连理工大学计算数学博士，卡内基梅隆大学机器人研究所联合培养博士，香港理工大学计算科学系博士后 (香江学者)。近年来在计算机视觉，深度学习，最优化方法等领域发表IEEE汇刊及CCF推荐A类会议论文70余篇，引用5000余次。成果获得教育部自然科学二等奖1项，辽宁省自然科学二等奖1项，6篇论文获得CCF推荐领域国际权威学术会议最佳 (学生) 论文类奖项，1篇期刊论文入选IEEE智能计算亮点论文。主持自然科学基金优青项目，科技部重点研发课题，入选辽宁省青年拔尖人才、百千万人才工程等。

个人主页：

https://rsliu.tech/

特别鸣谢本次Webinar主要组织者：

主办AC：苏航 (清华大学)

协办AC：刘日升 (大连理工大学)