VALSE 论文速览 第224期：弥合低秩适配和正交微调差异的Householder反射适配方法 ... ...

论文题目：

Bridging the Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation

作者列表：

袁深 (中国人民大学)，刘浩甜 (中国人民大学), 许洪腾 (中国人民大学)

B站观看网址：

https://www.bilibili.com/video/BV1o6o7YZEVS/

论文摘要：

While following different technical routes, both low-rank and orthogonal adaptation techniques can efficiently adapt large-scale pre-training models in specific tasks or domains based on a small piece of trainable parameters. In this study, we bridge the gap between these two techniques, proposing a simple but effective adaptation method based on Householder reflections. Given a pre-trained model, our method fine-tunes its layers by multiplying each frozen weight matrix with an orthogonal matrix constructed by a chain of learnable Householder reflections (HRs). This HR-based orthogonal fine-tuning is equivalent to an adaptive low-rank adaptation. Moreover, we show that the orthogonality of the reflection planes corresponding to the HRs impacts the model capacity and regularity. The analysis motivates us to regularize the orthogonality of the HRs, leading to different implementations of the proposed Householder reflection adaptation (HRA) method. Compared with state-of-the-art methods, HRA achieves superior performance with fewer learnable parameters when adapting large language models and conditional image generators. 

参考文献：

[1] Shen Yuan, Haotian Liu, Hongteng Xu. Bridging the Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation. NeurIPS2024 (Spotlight)

论文链接：

https://arxiv.org/abs/2405.17484

代码链接：

https://github.com/DaShenZi721/HRA

视频讲者简介：

Shen Yuan is a third-year Ph.D. candidate, under the supervision of Associate Professor Hongteng Xu, at Gaoling School of Artificial Intelligence, Renmin University of China. He received his undergraduate degree from the School of Computer Science and Engineering, University of Electronic Science and Technology of China. His research interests include large language models architecture design and parameter-efficient fine-tuning. From February to August 2022, he interned at Tencent AI Lab, where he co-authored a paper on the issue of imbalanced distribution in drug datasets under the guidance of mentors Lanqing Li and Peilin Zhao. From July 2023 to February 2024, he interned at BioMap, successfully completing his internship tasks. He has contributed to multiple research projects, including the Tencent AI Lab Rhino-Bird Focused Research Program and a Major Research Plan of NSFC.

个人主页：

https://scholar.google.com/citations?user=13aLBpkAAAAJ&hl=zh-CN

特别鸣谢本次论文速览主要组织者：

月度轮值AC：屈靓琼 (香港大学)