VALSE 论文速览 第207期：为图像问答任务构建好的上下文序列

论文题目：

How to Configure Good In-Context Sequence for Visual Question Answering    

作者列表：

李立 (东南大学)、彭嘉炜 (东南大学)、陈慧仪 (东南大学)、高重阳 (西北大学)、杨旭 (东南大学，通讯)    

B站观看网址：

https://www.bilibili.com/video/BV1JsK5eyEnr/    

复制链接到浏览器打开或点击阅读原文即可跳转至观看页面。

论文摘要：

Inspired by the success of Large Language Models in dealing with new tasks via In-Context Learning (ICL) in NLP, researchers have also developed Large Vision-Language Models (LVLMs) with ICL capabilities. However, when implementing ICL using these LVLMs, researchers usually resort to the simplest way like random sampling to configure the in-context sequence, thus leading to sub-optimal results. To enhance the ICL performance, in this study, we use Visual Question Answering (VQA) as case study to explore diverse in-context configurations to find the powerful ones. Additionally, through observing the changes of the LVLM outputs by altering the in-context sequence, we gain insights into the inner properties of LVLMs, improving our understanding of them. Specifically, to explore in-context configurations, we design diverse retrieval methods and employ different strategies to manipulate the retrieved demonstrations. Through exhaustive experiments on three VQA datasets: VQAv2, VizWiz, and OK-VQA, we uncover three important inner properties of the applied LVLM and demonstrate which strategies can consistently improve the ICL VQA performance.    

参考文献：

[1] Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, Xu Yang, “How to Configure Good In-Context Sequence for Visual Question Answering,” in Proceeding of IEEE Computer Vision and Pattern Recognition (CVPR 2024).

论文链接：

[https://arxiv.org/abs/2312.01571]

代码链接：

[https://github.com/GaryJiajia/OFv2_ICL_VQA]

视频讲者简介：

李立，东南大学软件学院硕士研究生，师从杨旭副教授，主要研究方向是多模态学习，上下文学习和大模型生成。

特别鸣谢本次论文速览主要组织者：

月度轮值AC：高广谓 (南京邮电大学)