VALSE 论文速览 第114期：An Investigation into Whitening Loss for SSL

论文题目：An Investigation into Whitening Loss for Self-supervised Learning

作者列表：翁熙 (北京航空航天大学)、黄雷 (北京航空航天大学)、赵磊 (北京航空航天大学)、Rao Muhammad Anwer (穆罕默德·本·扎耶德人工智能大学)、Salman Khan (穆罕默德·本·扎耶德人工智能大学)、 Fahad Khan (穆罕默德·本·扎耶德人工智能大学)

B站观看网址：

https://www.bilibili.com/video/BV1hM4y1E7ds/

论文摘要：

A desirable objective in self-supervised learning (SSL)is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW)based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations.

论文信息：

[1] Weng X, Huang L, Zhao L, et al. An Investigation into Whitening Loss for Self-supervised Learning[C]. Advances in Neural Information Processing Systems, 2022.

论文链接：

[https://arxiv.org/abs/2210.03586]

代码链接：

[https://github.com/winci-ai/CW-RGP]

视频讲者简介：

翁熙，本科毕业于北京航空航天大学人工智能研究院，目前为该学院研一学生，师从黄雷副教授。主要研究方向为自监督学习，以一作身份在NeurIPS 2022发表论文一篇。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：林迪 (天津大学)、彭春蕾 (西安电子科技大学)