Video classification via relational feature encoding networks

Abstract

© 2017 Association for Computing Machinery. In this paper, we propose a novel Relational Feature Encoding Network for video classification. The proposed network uses a set of relational functions wired on top of a backbone convolutional neural network (ConvNet) to generate multiple complementary feature streams on the fy, which are then combined by an aggregation module to form a video-level representation for recognition. The relational functions compute new relational features by applying element-wise operations or a simple projection to pairs of raw ConvNet features, and thus encode the underlying temporal dynamics and relationship of contextual frames which are critical for recognizing video contents. In this work, we explore a number of design choices for both the relational functions and the aggregation functions, and evaluate the resulting deep model on a number of video classification benchmarks, including the extended Fudan-Columbia Video dataset, UCF101, and Kinetics. Experimental results demonstrate that our model is not only well-suited for action recognition, but also exhibits promising performance for general videos.