Data-driven high-quality 3D model creation from sparse input

Abstract

Creating high-quality 3D models interactively on the basis of a user interface or through reconstruction from real captured data usually needs to carry out the challenging task of recovering geometric information from sparse input data, for example, a hand-drawn sketch, an incomplete point cloud or images with sparse views. Using data-driven strategies, in this thesis, we present novel algorithms for three problems: creating 3D face and caricature models from hand-drawn sketches, recovering the missing parts of an incomplete point cloud and performing human model reconstruction from sparse uncalibrated views. In the first part, a deep learning based sketching system is proposed for 3D face and caricature modeling. The system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. The network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. The system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that the sketching system can help users create face models quickly and effectively. To recover missing parts of 3D shapes, a new deep learning architecture is proposed which consists of two sub-networks: a global structure inference network and a local surface refinement network. The global structure inference network incorporates a Long Short-Term Memorized Context Fusion (LSTM-CF) module that encodes shape structure when given multi-view input depth information. It also incorporates a 3D fully convolutional (3DFCN) module that further enriches the shape structure representation by exploiting input volumetric information. Under the guidance of the global structure network, the local surface refinement network takes as input local shape patches around missing regions, and gradually outputs a high-resolution, complete surface through an encoder-decoder volumetric architecture. The method jointly trains the global structure inference and local surface refinement networks in an end-to-end manner. Qualitative and quantitative evaluations on six object categories demonstrate that our method outperforms state-of-the-art prior works. Finally, a novel two-stage algorithm is presented for reconstructing 3D human models wearing regular clothes from sparse uncalibrated views. The first stage reconstructs a coarse model with the help of a template model for human figures. A non-rigid dense correspondence algorithm is applied to generate dense correspondences. We fit the template model to the point cloud reconstructed from dense correspondences while enclosing it with the visual hull. In the second stage, the coarse model from the first stage is refined with geometric details, such as wrinkles, reconstructed from shading information. To successfully extract shading information for a surface with nonuniform reflectance, a hierarchical density based clustering algorithm is adapted to obtain high-quality pixel clusters. The algorithm has been validated with images from an existing dataset and images captured by a cell phone camera.