Real-time 3D surface tracking and its applications

We present a general technique for directly estimating and tracking surfaces from a stream of rectified stereo pairs in real-time. These techniques are based on the iterative updating of surface representations directly from image information and use no disparity search except during initialization. We perform the tracking through an iteratively re-weighted least squares minimization wherein a mask is incorporated to increase robustness to occlusion. The algorithms are formulated for a general family of linear in parameters surface models and discussed for the cases of planar surfaces and tensor product surfaces. These algorithms have been implemented on standard hardware and run at or near frame rate, with accuracy on the order of 1/20 of a pixel. We discuss applications of the technique including mobile robot localization, general deforming surface tracking, and biometry of biological surfaces.