Rosanswers logo

Update: Patrick Goebel has posted a great new package and tutorial for robust face detection. I have posted an answer to this question below (See Pi Robot example of robust face detection with Kinect:) if you find Patrick's work helpful, please vote up the answer to move it to the top of this question. (Scott)

I am moving this thread to the Answers site. It was started on ros-kinect before Answers was released, so trying to be a good ROS citizen. :)

Hi Radu I am able to pretty easily do face detection via the kinect using the built in recognition python package that comes inside the vision_opencv stack. For those on this thread here are the steps... roscd to openni_camera go into the launch directory, edit the openni_kinect.launch file and add the following line to remap the default video output topic from kinect to the topic that the face recognition listens node listens on.

Run the face recognition node with.. rosrun opencv_tests rosfacedetect.py (located in vision_opencv stack)

If performance lags a bit, use rosrun dynamic_reconfigure reconfigure_gui to adjust settings (I tend to use the QVGA options)

For any newbies reading this, you can use rosrun rviz rviz and add a pointcloud2 option that listens on /camera/depth/points2 topic to see the point cloud.

OK....So where does this get us? I now have the face found and boxes to a region of interest and can visualize it. I can capture and see the point cloud data of the larger scene. What I want to do to feed this into your recognition flow Radu is to then use the detected face ROI to crop the point cloud down to just the points that represent the face.

...and that is where I am stuck... Any tips from anyone on how to use ROI info to crop a point cloud image on the fly would be greatly appreciated.

And for those that are interested in facial recognition, hopefully this helps you get started.

Best Regards, -Scott

Originally posted by Scott on ROS Answers with karma: 693 on 2011-02-24

Post score: 11

Original comments

Comment by Asomerville on 2011-05-11:
Is this a question?

Comment by Tully on 2011-02-24:
FYI: you can edit an existing answer instead of reposting. I'm going to delete your duplicate question with they typo in the name.


3 Answers 3


Rosanswers logo

Scott, I've been keeping an eye on your thread on ros-users.

I've written a small package "face_builder" to detect faces using OpenCV and return a clustered point cloud of the face. It works as a stand-alone node, but it should also work as a library. It publishes clustered faces on the "faces" topic (type: sensor_msgs/PointCloud2) and publishes a "detected_faces" image for debugging.

It builds against trunk of PCL right now and ROS Diamondback. If you need to patch it to get it to build, let me know.

Not having a great place to put it, I uploaded it to the "PR" wiki: http://pr.willowgarage.com/wiki/Useful_general_references Look for "face_builder.tar"

Let me know what you find with face recognition.

Originally posted by watts with karma: 584 on 2011-02-27

This answer was ACCEPTED on the original site

Post score: 4

Original comments

Comment by Scott on 2011-05-17:
Hi Murph, glad to see your post. I have been off in rgbdslam/kinect/octomap land and my progress there has hit a bit of a wall. So in the interim, I am coming back to put in some more cycles on this as well. Please post your progress and tips and I will do the same as I try to follow after you.

Comment by Murph on 2011-04-07:
Has any more work been done on your face_builder? I'm going to hack on it some.


Rosanswers logo

Pi Robot example of robust face detection with Kinect:

Patrick Goebel has just announced a fantastic new package for face detection. Please go to the following link to try this out yourself. It is very well documented and robust. If you find Patrick's work to be helpful, please vote up this answer to move it to the top of this very popular question.

Link to Pi Face Tracker work: http://www.ros.org/wiki/pi_face_tracker

Originally posted by Scott with karma: 693 on 2011-06-04

This answer was NOT ACCEPTED on the original site

Post score: 4


Rosanswers logo


Preface: Face recognition is moving fast these days, this entry on answers.ros alone has had over 2000 hits since I first posted it. This answer is my attempt at a simple, yet effective approach to face recognition that I hope some find useful. It is very much a proof of concept and I hope those that give it a try will extend it and share their improvements with me and others.

Overview: The 2d face recognition methodology I present here is an easy and straightforward implementation that combines a modified version of the pi_face_tracker (pi_vision package) with the objects_of_daily_use package.

The pi_face_tracker code is used to “find” a persons head, crop it down to an roi (region of interest) then publish that image over a rostopic (ie the modified ros2opencv.py file in the pi_face_tracker package).

Setup: Link to my modified files for pi_face_tracker and objects_of_daily_use_finder: http://code.google.com/p/2d-face-recognition-basic/downloads/list

1st install the pi_vision package (gives you pi_face_tracker and ros2opencv), install the objects_of_daily_use package and install any dependencies they require. http://www.ros.org/wiki/pi_face_tracker http://www.ros.org/wiki/objects_of_daily_use_finder

Backup the ros2opencv.py file located in the ros2opencv/src folder of the pi_vision package Download my modified version of the file and put it in the src folder, replacing the original version

Download the two launch files and put them in the launch folder of the objects_of_daily_use_finder package.

Create two directories in the objects_of_daily_use_finder directory. faces_database faces_images

Edit both the launch files to give the FULL path to both of these files (relative path with ~ will not work)

Acquisition and Training of face images: Run : roslaunch ros2opencv openni_node.launch This will launch the driver for the kinect

cd to the faces_images folder_recognize] example mkdir scott cd to that persons named directory (VERY Important otherwise you will have to move all of you images) Note: you will repeat this step for each new person you want to learn and recognize You do not need to rename the images, as they will all be referenced by the parent folder that has their name

Run: rosrun image_view image_view image:=/camera/rgb/image_color This will open image view which lets us view raw images from the kinect Have a person face the kinect While the image_view screen is active, right click your mouse. This will save a jpg image to the folder you started the image_view from Have the person turn to different angles and make different facial expressions and each time right click the mouse again to get a new image

Cropping the images: Install an image editing program (I used GIMP) Open each file one by one, select a rectangle around the head region, then select crop to selection. Resave the image

Training: Run: roslaunch objects_of_daily_use_finder 2d_build_faces_db.launch This will do the training and will place the files needed for recognition in the faces_database directory you created earlier (as long as you properly edited the path in the launch files before running this step)

Detection/Recognition: Now for the fun part. Start the pi_face_tracker code: roslaunch pi_face_tracker face_tracker_kinect.launch Start the objects_of_daily_use_finder code: roslaunch objects_of_daily_use_finder 2d_detect_faces.launch

Usage: Currently, the code sets the roi for the face tracking when you draw a rectangle around the head of the person in the pi_face_tracker interface. You will see another small window pop-up that shows you what is being published to the objects_of_daily_use_finder. All of the code that affect what is sent to the objects_of_daily_use_finder is in the ros2opencv.py file You will see another window open which is for the objects_of_daily_use_finder. It will find what it thinks is the best match in the database for the image it is receiving Note: The attempted matching is continuous, even if you are not feeding it a proper image of a persons head) To determine if you have a good match or not you can examine the rostopic outputs from the objects_of_daily_use_finder to see the match confidence level, which ranges from 0 (poor match) to 2 (good match). Try it out and see what levels are high enough for you to consider it a match. For further details, see the pi_face_tracker (pi_vision) wiki and the objects_of_daily_use_finder wiki on ros at: http://www.ros.org/wiki/pi_face_tracker http://www.ros.org/wiki/objects_of_daily_use_finder

Opportunities to improve the code: Any methodology that allows you to identify a face (head), crop to that image (roi) and then publish it on a topic will work as a feed into the objects_of_daily_use_finder code. Possible other options would be to use the openni skeleton tracker to find a person and then use that information to set an roi as a way to crop the image to the head as well. Also, the current code does not “move” the roi as the person moves their head, so if they move around, you have to draw a new rectangle around their head in the pi_face_tracker gui in order to adjust the roi.

Questions and Feedback: Please use answers.ros and ask questions as a comment on this thread if you run into any problems so others can benefit. You can also e-mail me directly at [email protected]

If you make enhancements, please share with me and the others here so we can all benefit from the opensource model. If you like this answer, and find the code to be useful, please vote it up! Best Regards, -Scott

Originally posted by Scott with karma: 693 on 2012-04-10

This answer was NOT ACCEPTED on the original site

Post score: 2

Original comments

Comment by Pouyan on 2012-04-12:
For 2D face_recognition, you can also use the following package http://www.ros.org/wiki/face_recognition


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.