# How can computer vision distinguish one object being contained by another vs being on top of it?

How do we know that an object is contained inside another object or is just lying on top of it?

Lets take an example of a cup-plate-spoon. The cup is lying on top of the plate. But the spoon is inside the cup. How do we distinguish between the 2 situations? What are the criteria to decide whether A is contained inside B or just lying above B?

I am trying to solve it using kinect.

Actually we don't. This is the source of myriad visual illusions. Through a life time of experience we learn context which tells us when one thing can be on another vs. in it. But even then a sculpture can be built to trick us. For example it can look like a plate, cup, and spoon organized in a certain way but in fact be non of the above. A good example of this are the 3D printed Escher pieces.

The Kinect has the added ability to sense depth. With such information you may be able to make better determinations but we can still conceive of structures that would trick the system. I cannot speak to what CV methods exist to assist in your endeavor but I know that Deep Learning from the ML world has been making inroads here.

• Interesting. Are there not image recognition algorithms that account for depth? If it can see that there is a spoon and a cup, I would think that sufficiently precise distance sensor data and sufficient understanding of the 3D shape of the cup would allow for that to be determined. Commented Dec 22, 2012 at 22:53
• Like I said, I can't speak to the CV methods available. I'm inclined to agree that if we had sufficiently accurate depth data and knew that we were looking at a cup and a spoon then we could make the determination. The crux is knowing that we're looking at a cup and a spoon. How do we know as much? If we are just analyzing a single photo then we cannot definitively say that is the case. Even we humans we can only say what we think we see if we don't have a priori knowledge. Commented Dec 22, 2012 at 23:03

The problem is much easier, when you want solve this for specific object categories like cup-plate-spoon instead of any generic pair of objects. Also in my opinion there degeneracies as mentioned by DaemonMaker is not going to happen in this case since we have depth image instead of just a 2D image.

When you know the objects in scene and you have a depth image, it is possible to estimate the full 3D orientation (pose) of each of those objects. If the objects in question are rigid, some sort of ICP may be enough in getting this pose. For harder scenarios where the object is non-rigid (humans) or you don't know the exact specific model of the object (say tea cup vs coffee cup), you should refer to some computer-vision/Kinect literature.

Once you have the 3D poses of the objects, answering object-object 3D positional queries will be easier.

• Well, sorry for the very late response... I tried using convex hull method, I found the convex hull of both the regions in 3D. Then, found the amount of intersection they have. If it is significantly high, then one object is lying inside the other. If it is not, then it may be on top of it, or may not be. So I tested that by finding the 2D convex hulls of both the object regions, found the intersection. This gave me if one object lies on top of the other. Is there any method better than this? Commented Jul 27, 2013 at 11:53

I tried using convex hull method, I found the convex hull of both the regions in 3D. Then, found the amount of intersection they have. If it is significantly high, then one object is lying inside the other. If it is not, then it may be on top of it, or may not be. So I tested that by finding the 2D convex hulls of both the object regions, found the intersection. This gave me if one object lies on top of the other.