How to obtain dense point clouds from stereo cameras?

Question

I am trying to use a stereo camera for scene reconstruction, but I can usually only obtain sparse point clouds (i.e. over half the image does not have any proper depth information).

I realize that stereo processing algorithms rely on the presence of texture in the images and have a few parameters that can be tweaked to obtain better results, such as the disparity range or correlation window size. As much as I tune these parameters, though, I am never able to get results that are even remotely close to what can be obtained using an active sensor such as the Kinect.

The reason why I want that is because very often point clouds corresponding to adjacent regions don't have enough overlap for me to obtain a match, so reconstruction is severely impaired.

My question to the Computer Vision experts out there is the following: what can I do to obtain denser point clouds in general (without arbitrarily modifying my office environment)?

Answer 1

You can try to skip the salient point detection, and just densely sample over the image (as grid or so) and compute a feature descriptor at every sample point. You can probably even go as far as computing a descriptor for every pixel.

You might lose scale-invariance, but I think this won't hurt too much for stereo vision as objects will be at approximately the same scale in both images.

Another approach might be to combine multiple salient point detection algorithms: corners, edges, blobs and so on. Then you have to use the same feature descriptor algorithm for all detected points, however this latter part could be somewhat tricky to implement.

Answer 2

So the stereo image processing algorithms I have used in the past were implemented pixel by pixel. We just used the pinhole camera model and did some old fashioned measurements with measuring tape until our depth estimations matched the real thing.

The equations for a pair of parallel cameras are:

• $d = $half the distance between the cameras
• $f = $the focal length of the cameras (assumed to be the same)
• Coordinate Frames:
  • $x, y, z = $ coordinate frame between the cameras (i.e. the camera base frame)
  • $u_R, v_R$ camera coordinates in the right camera from the perspective of the robot (u is horizontal, v is vertical)
  • $u_L, v_L$ camera coordinates in the left camera
  • Note: the camera coordinates have their origins at the coordinate frame between the cameras (i.e. the u axes face opposite directions)

$u_L = \frac{f(x-d)}{z}$, $u_R = \frac{f(x+d)}{z}$

$zu_R = f(x+d)$, $zu_L = f(x-d)$

$z(u_R - u_L) = 2df$

$z = \frac{2df}{u_R - u_L}$

$y = \frac{v_L*z + df}{f}$

$x = \frac{u_L*z + df}{f}$

Using these equations you can compute a dense stereo cloud. One for each pixel on your cameras.

Answer 3

When you say, "over half the image does not have any proper depth information", which half ?

One issue we ran into is that if distance-to-object is of the same order of magnitude than your baseline (usually associated with very wide angle cameras), then the "standard" dense stereo algorithms don't work so well. We've been using the libelas library, and its developers told us that this is called "large baseline stereo" and is yet another problem.

Answer 4

Have a look at the KITTI Stereo Benchmark leaders. This benchmark penalizes algorithms for not producing a disparity at any given point, so the top performers produce dense outputs (even if they don't have much grounds for their estimate at many points). Some methods have publicly available code that is linked, which you can try out.

Note that they are not penalized for taking a long time, so many methods will take on the order of minutes per image to run and may not be suitable for your application. There have been many CNN-based methods introduced though that do very well, while still running in less than a second (on a GPU). At least one of these (CRL) has public code.

If you still don't get decent depth estimates with these on your images, then you may just be seeing the limits of what's feasible from a visible approach. There fundamentally needs to be distinctive texture in the area to be able to match points well, or you need a basis for making some strong assumptions (e.g., smoothness) in textureless regions.