I am currently working on a SLAM-like application using a variable baseline stereo rig. Assuming I'm trying to perform visual SLAM using a stereo camera, my initialization routine would involve producing a point cloud of all 'good' features I detect in the first pair of images.
Once this map is created and the cameras start moving, how do I keep 'track' of the original features that were responsible for this map, as I need to estimate my pose? Along with feature matching between the stereo pair of cameras, do I also have to perform matching between the initial set of images and the second set to see how many features are still visible (and thereby get their coordinates)? Or is there a more efficient way of doing this, through for instance, locating the overlap in the two points clouds?