VALSE 论文速览 第144期：AutoAD: Movie Description in Context

论文题目：AutoAD: Movie Description in Context

作者列表：

Tengda Han (VGG, University of Oxford), Max Bain (VGG, University of Oxford), Arsha Nagrani (VGG, University of Oxford), Gül Varol (École des Ponts, Univ Gustave Eiffel, CNRS), Weidi Xie (Shanghai Jiao Tong University), Andrew Zisserman (VGG, University of Oxford).

B站观看网址：

https://www.bilibili.com/video/BV1DB4y1R78P/

论文摘要：

The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. Generating high-quality movie AD is challenging due to the dependency of the descriptions on context, and the limited amount of training data available. In this work, we leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation. In order to obtain high-quality AD, we make the following four contributions: (i) we incorporate context from the movie clip, AD from previous clips, as well as the subtitles of the current shot; (ii) we address the lack of training data by pretraining on large scale datasets, where visual or contextual information are unavailable, e.g. text-only AD without movies or visual captioning datasets without context; (iii) we improve on the currently available AD datasets, by removing label noise in the MAD dataset, and adding character naming information; and (iv) we obtain strong results on the movie AD task compared with previous methods.

论文信息：

[1] Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, Andrew Zisserman. “AutoAD: Movie Description in Context”, in Proceeding of IEEE Computer Vision and Pattern Recognition (CVPR 2023), June 2023.

视频讲者简介：

Tengda Han is a post-doctoral research fellow at the Visual Geometry Group at the University of Oxford. He obtained his PhD from the same group in 2022 supervised by Andrew Zisserman. His current research focuses on self-supervised learning, efficient learning and video understanding.

个人主页：

https://tengdahan.github.io/

特别鸣谢本次论文速览主要组织者：

月度轮值AC：秦杰 (南京航空航天大学)

季度轮值AC：叶茫 (武汉大学)