Suggestions on object types (features) to track from ARDrone 2 camera

Question

UPDATE I have aded 50 bounty for this question on the StackOverflow

I am trying to implement object tracking from the camera(just one camera, no Z info). Camera has 720*1280 resolution, but I usually rescale it to 360*640 for faster processing.

This tracking is done from the robots camera and I want a system which would be as robust as possible.

I will list what I did so far and what were the results.

1. I tried to do colour tracking, I would convert image to hsv colour space, do thresholding, some morphological transformations and then find the object with the biggest area. This approach made a fair tracking of the object, unless there are no other object with the same colour. As I was looking for the max and if there are any other objects bigger than the one I need, robot would go towards the bigger one
2. Then, I decided to track circled objects of the specific colour. However, it was difficult to find under different angles
3. Then, I decided to track square objects of specific colour. I used this

       // Approximate contour with accuracy proportional
        // to the contour perimeter
        cv::approxPolyDP(
                cv::Mat(contours[i]),
                approx,
                cv::arcLength(cv::Mat(contours[i]), true) * 0.02,
                true
        );

and then I checked this condition

if (approx.size() >= 4 && approx.size() <= 6)

and afterwards I checked for

solidity > 0.85 and aspect ratio between 0.85 and 1.15

But still result is not as robust as I would expect, especially the size. If there are several squares it would not find the needed one.

---

So, now I need some suggestions on what other features of the object could I use to improve tracking and how? As I mentioned above several times, one of the main problems is size. And I know the size of the object. However, I am not sure how I can make use of it, because I do not know the distance of the object from the camera and that is why I am not sure how to represent its size in pixel representation so that I can eliminate any other blobs that do not fall into that range.

UPDATE

In the third step, I described how I am going to detect squares with specific colour. Below are the examples of what I am getting.

I used this HSV range for the red colour:

Scalar(250, 129, 0), Scalar(255, 255, 255), params to OpenCV's inRange function

HMIN = 250, HMAX = 255; SMIN = 129, SMAX = 255; VMIN = 0, VMAX = 255; (Would like to see your suggestions on tweaking this values as well)

So, in this picture you can see the processing; gaussian blurring (5*5), morphological closing two times (5*5). And the image with the label "result" shows the tracked object (please look at the green square).

On the second frame, you can see that it cannot detect the "red square". The only main difference between these two pics is that I bended down the lid of the laptop (please look closer if you cannot notice). I suppose this happens because of the illumination, and this causes the thresholding to give not desired results.

The only way, I can think of is doing two separate processing on the image. First, to do thresholding based on the colour as I was doing above. Then if I find the object to move to the next frame. If not to use this opencv's find squares method.

However, this method will involve doing too much of processing of the image.

Answer 1

I will attempt to answer slightly more than your question in order to expose you to alternative tracking methods. Note that what you are doing is essentially trying to locate and classify a certain object in an image. To do this robustly generally you need to have some training images and a label image that has each pixel labeled 0 or 1, where the pixels that are 1 are the object to be tracked. This is often done by hand. The training and label images are used to calculate a histogram of all the features of the object of interest, and those histograms are used either for simple range-based thresholding, or for use in a machine learning algorithm.

The following rely on having somehow located a region of interest that you want to describe. These ROIs may have been found by using the opencv::findContours function, or even randomly, or by testing a [100x100] window in a grid of every [60x60] pixels.

Shape descriptions: Area, perimeter, convexity, centroid, mean and standard deviation of distance of each pixel from centroid, skeleton length

Colour descriptions: mean, standard deviation, range, skew and kurtosis of intensities.

Texture descriptions: gabor filters response, standard deviation of intensities (in a local window)

Most of these are implemented in the python library scikit-image in the measure module.

In any case you will need to do the above on a 1-channel image. That may be just a single R/G/B channel, or combined into a grayscale channel, or for instance the (R - 0.5*(G + B)) channel if you were tracking for instance red apples.

Regardless of your detection method, it would be useful to track the object in pixel space with a Kalman Filter (KF). This would then give you additional spatial features which would be the difference between each contour centroid, and the expected position of the contour that the KF estimated.

Answer 2

You could use ArUco fiducials to demonstrate robust object detection and tracking. From there, you might be able to adapt the fiducial pipeline to detect more generic objects like colored squares.