VALSE 论文速览 第145期：ConZIC: Controllable Zero-shot Image Captioning

论文题目：ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing

作者列表：

曾泽群 (西安电子科技大学)， 张昊 (西安电子科技大学)，王正珏 (西安电子科技大学)，鲁瑞颖 (西安电子科技大学)，王东升 (西安电子科技大学)，陈渤 (西安电子科技大学)

B站观看网址：

https://www.bilibili.com/video/BV19c411d7u9/

论文摘要：

Zero-shot capability has been considered as a new revolution of deep learning, letting machines work on tasks without curated training data. As a good start and the only existing outcome of zero-shot image captioning (IC), ZeroCap abandons supervised training and sequentially searches every word in the caption using the knowledge of large-scale pre-trained models. Though effective, its autoregressive generation and gradient-directed searching mechanism limit the diversity of captions and inference speed, respectively. Moreover, ZeroCap does not consider the controllability issue of zero-shot IC. To move forward, we propose a framework for Controllable Zero-shot IC, named ConZIC. The core of ConZIC is a novel sampling-based non-autoregressive language model named GibbsBERT, which can generate and continuously polish every word. Extensive quantitative and qualitative results demonstrate the superior performance of our proposed ConZIC for both zero-shot IC and controllable zero-shot IC. Especially, ConZIC achieves about 5× faster generation speed than ZeroCap, and about 1.5× higher diversity scores, with accurate generation given different control signals.

论文信息：

[1] Zequn Zeng, Hao Zhang, Zhengjue Wang, Ruiying Lu, Dongsheng Wang, Bo Chen, “ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing,” CVPR 2023.

论文链接：

[https://arxiv.org/abs/2303.02437]

代码链接：

[https://github.com/joeyz0z/ConZIC]

视频讲者简介：

曾泽群，西安电子科技大学博士生，陈渤教授团队。研究方向为多模态学习，曾在计算机视觉顶会顶刊CVPR，IJCV发表工作。

个人主页：

https://joeyz0z.github.io/

特别鸣谢本次论文速览主要组织者：

月度轮值AC：谢雨彤 (阿德莱德大学)

季度轮值AC：张磊 (重庆大学)