How to select cameras for a stereo vision system?

Question

I am in the process of building a stereo vision system to be used on a UGV. The system is for a robot that will be used in a competition wherein the robot is teleoperated to find relatively small colored rocks in a large outdoor field. I understand how to calibrate such a system and process the data for a stereo vision system. I do not however know how to select cameras for such a system. What are the best practices for picking cameras for a stereo vision system?

Answer 1

Off the top of my head I would go by the following selection criteria

• color/b&w - usually b&w is better, since the stereo algorithms only use one channel anyway
• baseline - this really depends on the scene. With the baseline you control how much accuracy you get with distance. Wider baseline results in higher disparity values and thus less noise on the distance estimation. Higher baseline also means that you will get a reduced overlap between your field of views. Most importantly though, the wider the baseline the harder the matching between the two views. So the quality of the result goes down.
• shutter - always use a global shutter for anything with computer vision on mobile robots
• resolution - most stereo algorithms are computationally expensive. And you will likely not need that many 3d points. In the near field the sampling density is usually enough, and the far field your error from the low disparity is more of a problem than sampling density. 640x480 is fine in most cases. Some chips have a wider aspect ratio, which is favourable for the overlap, but you can get the same with using a sub-window of your chip.
• lens/field of view - for mobile outdoor robots I prefer a wide field of view over narrow field of view. More or less for the same reasons as the resolution. Make sure your lenses are rigid and have a means to fixate any moving parts e.g. like adjustable focal length. In most cases a fixed focal length is fine anyway, since you are limited by the selection of your baseline.
• trigger - when you create your own stereo rig, your cameras should support hardware trigger and ideally one should trigger the other.
• framerate - In my opinion the least important consideration. Unless you have a lot of processing power you won't be getting much more than something like 5 Hz anyway. So your algorithm is much more likely to be the bottleneck, rather than the chip or the connection.
• interface - Most cameras come with Firewire/USB2/USB3/Ethernet. USB2 is useless as it consumes a lot of CPU. Firewire is good if you have a port, but I think it seems to be on the decline in favour of USB3 these days. Didn't have much experience with USB3 yet. Ethernet is quite nice because you are more likely to find embedded systems with GigEthernet than USB3 at the moment.
• housing - as rigid as possible. Recalibrating is an annoying procedure.

Answer 2

A few things you should be on the lookout for:

• Global shutter basically means all pixels get captured at the same time, as opposed to Rolling shutter where they are captured sequentially in a line scan fashion. Since your UGV will be moving around and performing stereo algorithms over the images you capture, it could be important that you avoid aberrations that occure when camera move, such as the ones seen in the pictures below (taken from Wikipedia):

• Camera synchronization in hardware is achievable by some cameras, notably firewire cameras AFAIK. That can greatly improve the results for stereo when things are moving around.

• Mounting should be done in a way that changes in the extrinsic parameters (the relative pose between cameras) for the stereo pair will be unlikely to change after calibration. In your rig that might be more important, since the UGV might face uneven terrain outdoors and things will vibrate.

• Dedicated stereo hardware makes it possible to acquire disparity images directly as output of your stereo vision system, which eases the load off your embedded computing. It also tends to be much faster than running the very same algorithms in software.

As usual, the more you're willing to pay the better the results. To be honest, if you're able to buy a full-blown stereo camera such as the Bumblebee2, that's what I'd do. Otherwise, if you're on the cheaper side, I would simply go with a Kinect: it's unlikely you'll be able to get a system that outperforms it for the same price.

Answer 3

You should start by calculating how many frames per second you need, and how much camera resolution you can process at that framerate. If nothing else, that will prevent you from overspending or from buying a camera that won't suit your needs.

Beyond that, there are a variety of features that make the choice more difficult/interesting. Different cameras (especially network cameras like Axis) allow you to alter the image quality, or to specify a max bitrate for the image stream. Some cameras also give you choices on the shutter speed, allowing you to favor a constant exposure time or a constant average illumination in the image. Some cameras are more sensitive than others (the last time I worked with this was in 2009, and we noticed that the PS3 Eye did really well in low light conditions).

Probably the best thing you can do would be to run your image processing algorithms on a few static images that you take with a DSLR, then attempt to reduce the frame size and quality to see where things begin to break down.

Answer 4

That is an interesting topic, and not very easy to get right on the first try. From experience with this, here are the most important things.

• Synchronization. The camera must be 100% synced. For example, say the UGV is driving at a modest 36Km/hr (10m/s) and recording frames at 30 frames per second. That is, at every frame the UGV would cover 3m. Now, say your sync is off by 1ms, one of the cameras will be off by ~0.3m, which is bad [just off the top of my head]. Sync problems are very hard to detect.

• Resolution. In the context of stereo, we refer to depth resolution, or how much depth can we resolve given a change in disparity. An excellent explanation can be found here http://pub1.willowgarage.com/~konolige/svs/disparity.htm The equation you want to look at is a function of the focal length of the image and the stereo baseline. $\delta Z = \frac{Z^2}{Bf}\delta d$. This says that depth resolving power $\delta Z$ is inversely proportional to the baseline $B$ times the focal length $f$. That is the longer the baseline, the smaller $\delta Z$, the better. Meaning, you can resolve more depth. The minimum change in disparity $\delta d$ is controlled partially by the image resolution and the algorithm used. For your calculations, you can safely assume that $\delta d$ is at least $1/2$. It is better to be conservative with the choice of $\delta d$.

• Overlap. You want to have overlap between the cameras to get stereo. Hence, you need to choose a combination of lens focal length (field of view) and baseline so that you have enough overlap for the application. Basically, trigonometry work on the board, or a quick matlab/python script.

For UGV's, there are two uses for stereo.

1. Navigation and Pose Estimation In this case you most probably need a large baseline + long focal length. This allows the stereo to see and resolve depth better and longer range.

2. Obstacle detection and avoidance You will probably need a shorter baseline and wider lens (smaller focal length) so that you can focus on things very close to you.

Some UGV's might have both stereo setups, the large baseline narrow field of view for navigation and another one or two for obstacle avoidance.

Be very careful what you buy. Some companies offer already built stereo setups. Those are great on the robustness side of things, they don't loose calibration easily and are always in sync. The problem is that the commercially available ones have a small baseline. If you want to build your own. I'm guessing you will end up doing so, make sure the camera are syncable. Firewire is great for this, two camera on the same bus will sync with 125microsecond accuracy out of the box! USB and Gige cams are painful to sync. When you put everything together, you want to make sure that the lenses are not going to move at all, and the baseline is rigid, very rigid for the application.

Also be careful in lens selection. Not all lenses work with all cameras. Lenses have a resolution too. This is a another topic, here is a short article on this http://www.qualitymag.com/articles/90642-q-a--selecting-lenses-for-machine-vision-systems