I want to learn robotics and really interested in making a robot based on Kinect sensor.
I see so many projects like this one, and just wondering how it works on top level. I downloaded Kinect SDK and did some basic tutorials, but I just don't think that Microsoft SDK is the library to use for real robotics projects. Any suggestions where to start and what library to use? Any good books in particular or online resources?
Welcome to the Robotics StackExchange! The Kinect is only a sensor, albeit a relatively powerful one for its price. To use it with a robot, you need to cover basically two bases:
Get a robot that is able to interface with it
That means you need to somehow build/buy something consisting of processing unit (a microcontroller or PC) with USB connectivity and the appropriate drivers to recognize and control the Kinect (usually easier with a PC running Linux). This thing also needs to be able to provide appropriate power input to the sensor.
You also want it to be able to move, so you need to add motors or other actuators to it and figure out how to get your processing unit to control them (usually easier with a microcontroller).
If you want it to be able to process all the data from the Kinect and make complex decisions by itself you will need enough processing power onboard (usually a PC). Otherwise, you might have to add wireless communication with other computers that will do the heavy calculations.
Figure out how to get the Kinect data and what to do with it
The Microsoft SDK is perfectly usable, as is OpenNI or libfreenect... It really comes down to what operating system is running on your robot. I am not aware of how to interface the Kinect with less powerful microcontrollers not running a popular OS (Windows/Linux/Mac) and while it may be possible it definitely is not desirable (as in it would be a lot of work for a beginner).
Once you've chosen which programming library and driver to use to get Kinect data, you will need to find out what you want to do with the data you get. The raw sensor data is just a range image, but some of these libraries can provide some higher-level info extracted from these range images such as skeletonized models of people.
In short:
Start by figuring out what kind of robot you want, how you want it to move, where the Kinect will be placed and so on.
Choose your processing unit after searching for alternatives that are able to interface with the Kinect easily. I would suggest something like a netbook running Linux to a beginner, with a microcontroller board like the Arduino connected via USB to control motors and other low-level components of the robot.
Once you have the basic components of your robot ready (e.g. mechanical parts in place, PC running Linux, motor control working, etc) add the Kinect to the mix by connecting it to your processing unit and start working on what to do with it.
Excellent answer by George. If I may, I'd like add more details and suggestions. (I would recommend googling terms that are new to you).
Your entire robot (input) configuration depends on what kind of data your code is going to process. If it's more vision oriented, the data can be depth maps, point clouds, rgb images or a combination of the 3. This means you can use something like the Kinect (which would require a more powerful processor that the arduino), or simple rgb cameras (for which something like the arducam+arduino is sufficient). Arduino can run simple computer vision programs, but nothing too heavy and processor intensive. Also, if I'm not wrong, Arduino cannot process and use its GPIO pins at the same time. Which means during an intensive processing task, your robot motors are basically off.
In my experience, if you're planning to use point cloud stream, it would require a dedicated graphics hardware (like a GPU chip that's present on the Raspberry Pi or pcduino). (Hell, even my laptop cannot run the point cloud stream openni drivers without a dedicated graphics card)
Depth maps can be easily processed without a GPU, but it still requires high processing capabilities which would render the Arduino micro'controller' pretty useless. In this case you'd require a microprocessor board viz. Beagleboard, pcduino, gumstix, Raspi, Arduino TRE (not released yet), to name a few.
In terms of the output, the way to control your robot's mobility is to send power to the dc motors, servos, or stepper motors of your robot. This requires writing programs similar to what one would write on the microcontroller like an arduino.
If you require microprocessors, in my opinion (although I've never used it before), the pcduino V2 has it all: First of all, it's relatively cheap (~$70 + some cables etc). Raspi is cheaper, but not as powerful. Pcduino has 1Ghz A10 processor, expandable upto 32GB SD slot, a GPU chip (OpenGL ES2.0) which means you can program CV applications and integrate webcams, or maybe even a Kinect. It also has wifi arduino headers, HDMI and usb outputs for easily connecting to monitors for programming etc. But the thing I love about pcduino is that it has arduino headers. Which means you can attach something like the ADAfruit motor shield (normally used with arduino to effectively control a bunch of motors) on top of the pcduino board and send motor signals to control your robot.
Hope this helps
Evil Closet Monkey nicely itemized the possible Kinect libraries in his answer to another Kinect related question mentioning OpenKinect SDK and OpenNI SDK besides Windows SDK.
A good book on Kinect is Making Things See by Greg Borenstein from O'Reilly.
As a follow-up (albeit after 2.5 years) I would like to add some other things that I was able to work on and might possibly help you / anybody else foraying into this field.
Option 1) Use an existing robot hardware and add your own sensors and actuators:
For e.g.: I had access to a iRobot Create (Roomba without the vaccum cleaner). So I modified that hardware-wise by adding a long "neck" made out of a plastic tube that was lying around (I had to add a tough-plastic platform on top of the Create on the screws provided so that I could fix whatever I wanted to this plastic platform (see image below to get an idea).
You can then interface with the Roomba from your laptop or other PC using several Roomba interface packages available (I used the one for Matlab since my CV code was in Matlab). I connected a Kinect v2 to my laptop and thus I had complete control over the robot as well as was able to acquire data from the Kinect. The wires and electronics that you see on the robot are from my previous attempt to supply power directly from the robot battery to the Kinect v1 that I was using on it earlier to make the robot completely mobile. For the v2 I did not bother with the power conversion as it was complicated so I just used tethered power supply for the kinect v2. Since I had a laptop (Thinkpad with usb 3.0 that the Kinect v2 needs), the interfacing with Kinect v2 was relatively easy; I used the libfreenect2 for running Kinect v2. I called it as a system call from my Matlab image processing code.
The advantage of this system is that it's cheap. You only have to pay for the robot base, the Kinect, and any other hardware you buy. The disadvantage is that it's very slow moving robot since it's top heavy. The Kinect v2 also had to be tethered. This setup cost me around \$ 300 since the base was pre-owned from ebay, and the Kinect costs ~$200. Not including any additional hardware costs.
Option 2): Build your own robot hardware. This option is more time+money consuming. The way I did this was I purchased a 6-wheel drive robot chasis (it's not cheap; around $300) from sparkfun. To this I added a pan-tilt system from servocity. And to this PT system I added a asus xtion (since it's light). The motors of the robot and the PT system were controlled by a gorilla controller board, which received commands from a Jetson TK1 which was processing the Asus depth data, and sending commands to the motors. Unfortunately I do not have the pictures on my machine right now; but it looks a bit like a cross between Wall-E and a 6WD military vehicle ;) . Jetson TK1 is considerably powerful and I've run Kinect v1, Asus Xtion, and Kinect v2 on it (it has a usb 3.0 as well). For Kinect v1 and Asus I used OpenNI2 and for Kinect v2 I used libfreenect2 (for both these robot options I used Ubuntu btw). But since the PT system was present I stuck with the lightweight Asus sensor.
The advantage of this system is that you can change it however u want with different hardware or electronics. The disadvantage is that you've to do a lot of motor controller programming on your own which means you've to understand how to control motors using PWM signals, for example. This setup aint cheap. Cost me around $800-900 without the additional hardware I had to purchase to make it work.
In both these options I did not use any odometry data coming from the wheels or sensors (e.g. how many rotations of the wheel, data from the IR sensors on the Create etc). I relied entirely on vision data.
Hopefully this helps. Let me know if you have questions.
