Once matching features are computed between a stereo pair, how can they be tracked?

Question

I am currently working on a SLAM-like application using a variable baseline stereo rig. Assuming I'm trying to perform visual SLAM using a stereo camera, my initialization routine would involve producing a point cloud of all 'good' features I detect in the first pair of images.

Once this map is created and the cameras start moving, how do I keep 'track' of the original features that were responsible for this map, as I need to estimate my pose? Along with feature matching between the stereo pair of cameras, do I also have to perform matching between the initial set of images and the second set to see how many features are still visible (and thereby get their coordinates)? Or is there a more efficient way of doing this, through for instance, locating the overlap in the two points clouds?

Answer 1

This will depend a bit on how you are detecting your features and how you are estimating motion between frames. Classic approaches would start with detecting SIFT-like features in each image, which provide a descriptor for each detected feature as well as the point locations. These descriptors provide evidence of whether two detected features correspond to the same world point.

In these classical approaches, you will need to do a search for corresponding feature points between successive stereo image frames in order to estimate motion, as well as possibly within the stereo pair. Depending on how your frame rate compares to camera motion, you may be able to restrict the search area for these correspondences. Then this same correspondence information provides most of what's needed for tracking. See this Stereo Visual Odometry Tutorial for a detailed overview of such an approach.

There may be benefit to keeping track of particular points over longer periods as well, since longer-lived features show evidence of being more stable and could be less prone to be spurious matches. To take advantage of that, you need to perform some bookkeeping of things like the quality of matching over time or its agreement with estimated motion, which you can then use to decide how much to trust the point.

Also see the KITTI odometry leaderboards for a list of algorithms that do very well at this task. The papers linked for those entries should give a good idea of the variation in how leading approaches go about this. Note that some of these use stereo vision, and others use laser data.