Generalized rank conditions in multiple view geometry with applications to dynamical scenes

Abstract

In this paper, the geometry of a general class of projections from ℝn to ℝk (k < n) is examined, as a generalization of classic multiple view geometry in computer vision. It is shown that geometric constraints that govern multiple images of hyperplanes in ℝn, as well as any incidence conditions among these hyperplanes (such as inclusion, intersection, and restriction), can be systematically captured through certain rank conditions on the so-called multiple view matrix. All constraints known or unknown in computer vision for the projection from ℝ3 to ℝ2 are simply instances of this result. It certainly simplifies current efforts to extending classic multiple view geometry to dynamical scenes. It also reveals that since most new constraints in spaces of higher dimension are nonlinear, the rank conditions are a natural replacement for the traditional multilinear analysis. We also demonstrate that the rank conditions encode extremely rich information about dynamical scenes and they give rise to fundamental criteria for purposes such as stereopsis in n-dimensional space, segmentation of dynamical features, detection of spatial and temporal formations, and rejection of occluding T-junctions.