5
$\begingroup$

My task:

I have a task where I am asked to track parcels(carton boxes) of different dimensions moving on a conveyor. I am using Asus Xtion pro camera mounted on top of a conveyor in any inclined angle. I am looking for a model free object tracker that will detect boxes in the scene, track them & gives their 6 DOF? My target object is just a box and I want to eliminate all other things in the scene.

My approach:

  1. I do Point cloud pre-processing like down-sampling, pass through filtering and segmentation. All these should give me a final point cloud containing only the objects on the conveyor.

  2. I planned to make the "z" values in each point(depth value) as zero, thereby making the point cloud of the box to be flat on the ground.

  3. I planned to transfer the view of the camera from any inclined position to a top down view so that I can view any number of carton boxes moving on the conveyor from a top down view. I feel the top down view will prevent perspective viewing problems

The process flow of step 2 and 3 is shown below. enter image description here

  1. After the top down view of the point cloud is achieved, I need to convert the 3rd point cloud to 2nd image, so that I can perform object tracking with so many OpenCV based tracking algorithms available.

A Sample point cloud is shown below in different views

Original View from camera:

enter image description here

Point Cloud View 1: enter image description here

Point Cloud View 2: enter image description here

Point Cloud Target/Desired View for converting to 2nd: (The box is the target. All the ground plane and unnecessary points would be eliminated) enter image description here

Is my approach correct? How will I achieve steps 2,3 and 4?

Improve this question
$\endgroup$
8
  • $\begingroup$ If you know the scene when it is "empy" (i.e. no boxes coming on the conveyor) and it is failry static in the region of interest you can select that region of interesed based on an bounding box in 3D space, placed where you expect the boxes. You can delete all the rest. In the region of interest substract the static image and al you are left with is the box and noise. Filter, project to XY plane and there you have it... $\endgroup$ – 50k4 Oct 25 '16 at 9:21
  • 1
    $\begingroup$ This sounds like an open-ended design question, and the answer would be primarily opinion-based (is it correct, how can you achieve), but I'll let the community weigh in for a couple days before I decide to close it. That said, what is the point of flattening the point cloud along the z-axis? Don't you want to know "their 6 DOF?" If you're moving the view point to overhead I don't know what flattening gets you. $\endgroup$ – Chuck Oct 25 '16 at 18:04
  • $\begingroup$ @Chuck yeah i want to a have 6 DOF, but the problem is that, when we transform the point cloud without having z = 0, we will have image where side faces of the boxes would also be visible as the box is moving. $\endgroup$ – MaheshKumar Oct 25 '16 at 19:04
  • $\begingroup$ @50k4 yeah we can get only the boxes and noise as you said, but after projecting to the XY plane do we get top down view? $\endgroup$ – MaheshKumar Oct 25 '16 at 19:06
  • $\begingroup$ If you manage to get rid of every point from the point cloud, but the box then you get a sort of top down view. Depending on the viewing angle, obviously as close as you get to a actual top down view with your camera the better the projected coordinates will reflect a top down view. Since you mentioned that the box is moving, can you compensate for that? do you know the conveyor velocity? $\endgroup$ – 50k4 Oct 25 '16 at 19:24
0
$\begingroup$

It looks like you're using ROS which will make some of these tasks easier. To start, you will need to define some coordinate frames. If you're only using the coordinate frame of the lidar, that's fine, just make sure you know the location and orientation of your origin. It's important to be able to define the coordinate frames of other objects (like the boxes) in terms of relative transforms from other frames (like the lidar origin).

My first step in this process would be to eliminate all points which are irrelevant by knowledge of the physical system rather than through software processing. How high is the lidar off the ground? How high is the conveyor off the ground? You know that no points below the conveyor belt (or off to the sides for that matter) are going to be relevant and you know the location of the belt relative to the lidar - crop the field of view to match.

Next, consider what "flattening" the point cloud will actually gain you. Since the lidar seems to be above the plane of the top of the objects that you're looking to find, you will get a large number of points returned from both the sides and top of the objects. This means that things like edge detection won't work on just a 2D representation of that dataset. I would recommend going down another path of "known object" filtering. Something like volumetric estimation or plane detection/tracking might be your best bet. Hope this helps!

Improve this answer
$\endgroup$
-1
$\begingroup$

I cannot answer (and I don't think anyone would be able to) the question of "is this the correct method". It is subjective and, in the end, you can only try and see :).

But I can tell that this would not be the way I would go with this problem. Why ?

  • In my experience, point cloud processing, especially when trying to find shapes, is slow and hard. Since you're only searching for box and you have a 2D image, I would start there. Unless you know the background and can quickly remove it, I would not do step one.

  • I'm not sure why finding the top view will be of any help. If anything, it will be problematic if the boxes are stacked one above the other.

Another idea

Consider using deep learning (or just learning) on your 2D image to find the boxes and extract their pose from the point cloud once they have been found. If you need tracking after use some tracking algorithm on the 2D image.

Learning to recognize a box should be feasible given that you have a good training dataset. Then, once you have localized them in the 2D image, finding the pose is a matter of extracting it from the point cloud. Since you are using an Xtion, the point cloud and the 2D image should have a simple fixed transformation between them such that it should not be too hard to do that (hint: find the faces of the box).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could the downvoter explain the downvote on this answer ? :/ $\endgroup$ – Malcolm Aug 9 '18 at 13:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.