This is an excellent question that is currently being explored directly by field experts. Here are some of the latest publications that consider the problem you are encountering:
The first item here considers an interesting application of supervised learning. Since we can't rely on our object detection algorithm to classify our object, we can instead seek to learn high probability grasping points.
The other three papers I included may simply be of additional interest, but I won't get into their details here. I do strongly recommend giving them a read.
If you provide more details on your specific problem (i.e. where the camera is in relation to the hand, what is the view angle available by the camera, etc.) and let us know what you have tried/considered in detail, then I would be happy to give the specific problem a go and edit my answer accordingly (This likely should have been a comment, but I don't have that privilege yet).