VALSE 论文速览 第113期：Understanding the Failure of BN in Transformer

论文题目：Understanding the Failure of Batch Normalization for Transformers in NLP

作者列表：王嘉曦 (清华大学)、吴及 (清华大学)、黄雷 (北京航空航天大学)

B站观看网址：

https://www.bilibili.com/video/BV1KT411H73H/

论文摘要：

Batch Normalization (BN)is a core and prevalent technique in accelerating the training of deep neural networks and improving the generalization on Computer Vision (CV)tasks. However, it fails to defend its position in Natural Language Processing (NLP), which is dominated by Layer Normalization (LN). In this paper, we are trying to answer why BN usually performs worse than LN in NLP tasks with Transformer models. We find that the inconsistency between training and inference of BN is the leading cause that results in the failure of BN in NLP. We define Training Inference Discrepancy (TID)to quantitatively measure this inconsistency and reveal that TID can indicate BN's performance, supported by extensive experiments, including image classification, neural machine translation, language modeling, sequence labeling, and text classification tasks. We find that BN can obtain much better test performance than LN when TID keeps small through training. To suppress the explosion of TID, we propose Regularized BN (RBN)that adds a simple regularization term to narrow the gap between batch statistics and population statistics of BN. RBN improves the performance of BN consistently and outperforms or is on par with LN on 17 out of 20 settings, involving ten datasets and two common variants of Transformer.

论文信息：

[1] Wang, Jiaxi and Wu, Ji and Huang, Lei,“Understanding the Failure of Batch Normalization for Transformers in NLP,”In Advances in Neural Information Processing Systems, 2022.

论文链接：

[https://arxiv.org/abs/2210.05153]

代码链接：

[https://github.com/wjxts/RegularizedBN]

视频讲者简介：

王嘉曦，清华大学博士生。主要研究方向为理解深度学习中的归一化层, 机器学习用于药物发现。目前在ccf A/B类会议上发表两篇一作论文。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：林迪 (天津大学)、彭春蕾 (西安电子科技大学)