3D trajectory recovery in spatial and time domains from multiple images

Abstract

Recovering 3D structure from multiple 2D images is a fundamental problem in computer vision. Most of existing methods focus on the reconstruction of static points in 3D space; however, the reconstruction of trajectories which are resulted from moving points should also have our full attention due to its high efficiency in structure modeling and description. Depending on whether points are moving in spatial domain or in time domain, trajectory recovery turns out to be a curve reconstruction problem or a non-rigid structure recovery problem respectively. This thesis addresses several issues that were not considered in existing approaches in both of the two problems. For the curve reconstruction problem, we propose a dedicated method for planar curve reconstruction and an optimization method for general curve reconstruction. In the planar curve reconstruction method, measured projected curves that are typically represented by sequences of points are fitted using B-splines before reconstruction, enabling the occlusion problem to be handled naturally. Also, an optimization algorithm is developed to match the fitted curves across images while enforcing the planarity constraint, and the algorithm is guaranteed to converge. In the general curve reconstruction method, Non-Uniform Rational B-Spline (NURBS) is employed for curve representation in 3D space, which improves the flexibility in curve description while maintaining the smoothness of a curve at the same time. Starting with measured point sequences of projected curves, a complete set of algorithms are developed and evaluated, including curve initialization and optimization of the initialized curve by minimizing the 2D reprojection error that is defined to be the 2D Euclidean distance from measured points to reprojected curves. Experiments show that the proposed methods are robust and efficient, and are excellent in producing high-quality reconstruction results. For the non-rigid structure recovery problem, we proposed two methods for the recovery of non-rigid structures together with a strategy that automates the process of non-rigid structure recovery. Compared with existing methods using synthetic datasets, both of the two proposed methods perform significantly better than existing methods when there are noise contaminations in measurements, and are capable to recover the ground truth solution when the measurements are noise free whereas no existing method is capable of achieving this so far. In the first method, namely factorization-based method, the available constraints in non-rigid structure from motion are analyzed and the ambiguity of the solution space of the proposed method is clarified, leading to a straightforward approach that requires only solution to several linear equations in least-squares sense instead of having to solve non-linear optimization problems in existing methods. In the second method, namely bundle adjustment method, a modified trajectory basis model that is demonstrated to be more flexible for non-rigid structure description is proposed. The method seeks for optimal non-rigid structure and camera matrices by alternately solving a set of linear equations in least square sense. Experiments on real non-rigid motions show that the method improves the quality of reconstruction significantly.