Visual understanding from data-driven feature learning to causality modeling

Abstract

Human visual systems are complex. Teaching machines to see and understand like humans has always been a dream for us. One of the major challenges is visual understanding, a fundamental computer vision task that aims to grasp the visual perception ability of the human visual system. Although the performance of existing research in visual understanding has been significantly improved in the wave of deep neural networks, data-driven feature learning based methods still have many unresolved challenges. For example, models learned under data-driven feature learning paradigm tend to capture correlation instead of causation from data. Therefore, building models merely depending on data-driven feature learning could be problematic.

In this thesis, we aim to build better visual systems that could understand, interact with, and reason about the visual world. To conduct a comprehensive analysis of the problems of existing visual understanding models, I present four research works, each of which focuses on an important sub-task of visual understanding, from 2D image-level, 3D scene-level, to video-level visual understanding. Concretely, I identify three key problems of data-driven feature learning based visual understanding models. To bridge such research gaps, I propose to exploit data-driven feature learning as a building block together with tools such as causal inference and pre-training to construct more sensible and causality-aware human-level cognitive system.

In general, this thesis integrates approaches to feature representation learning, high-level perception, and causal and counterfactual reasoning in data-driven feature learning, forming a unified framework to build better machine intelligent visual systems. First, we introduce a complete feature description, called Complementary Parts Model, superior to existing data-driven feature descriptions. Second, we propose a Scene-Intuitive Agent to demonstrate that high-level understanding should be built upon data-driven feature learning. Then, we leverage Causal Inference to solve the data correlation problem in existing data-driven feature learning based visual understanding models.