VALSE 论文速览 第111期：Rethinking Attention-Model Explainability

论文题目：Rethinking Attention-Model Explainability through Faithfulness Violation Test

作者列表：刘一兵 (香港城市大学)、李皓亮 (香港城市大学)、郭洋洋 (新加坡国立大学)、孔臣祺 (香港城市大学)、李菁 (香港理工大学)、王诗淇 (香港城市大学)

B站观看网址：

https://www.bilibili.com/video/BV1tm4y1y73t/

论文摘要：

Attention mechanisms are dominating the explainability of deep models. They produce probability distributions over the input, which are widely deemed as feature-importance indicators. However, in this paper, we find one critical limitation in attention explanations: weakness in identifying the polarity of feature impact. This would be somehow misleading-features with higher attention weights may not faithfully contribute to model predictions; instead, they can impose suppression effects. With this finding, we reflect on the explainability of current attention-based techniques, such as Attention Gradient and LRP-based attention explanations. We first propose an actionable diagnostic methodology (henceforth faithfulness violation test)to measure the consistency between explanation weights and the impact polarity. Through the extensive experiments, we then show that most tested explanation methods are unexpectedly hindered by the faithfulness violation issue, especially the raw attention. Empirical analyses on the factors affecting violation issues further provide useful observations for adopting explanation methods in attention models.

论文信息：

[1] Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, Shiqi Wang, “Rethinking Attention-Model Explainability through Faithfulness Violation Test”. International Conference on Machine Learning (ICML), 2022.

论文链接：

[https://proceedings.mlr.press/v162/liu22i.html]

代码链接：

[https://github.com/BierOne/Attention-Faithfulness]

视频讲者简介：

Yibing Liu is currently a second-year Ph.D. student in the Department of Computer Science at City University of Hong Kong. Before that, he received my B.E. degree in School of Computer Science and Technology from Shandong University in 2020. His research focuses on understanding internal workings of various machine learning algorithms, and designing tools to make them explainable and robust.

特别鸣谢本次论文速览主要组织者：

月度轮值AC：林迪 (天津大学)、郑乾 (浙江大学)