I want to embed environmental data collected from sensors into a live video stream from a camera. Has anyone done this or know how I would go about doing something like this? Is there a library available for the arduino or RPi?


3 Answers 3


There is no easy way of doing it. I've done it in many of my projects, the best method I found was to use http://opencv.org/ to capture the frames, and overlay each frame with your data before displaying it again. The code below gives you an idea (written in the old opencv version 1 syntax)

* Display the video from webcam and overlay text

#include <stdio.h>
#include "cv.h"
#include "highgui.h"

int main(int argc, char** argv)

    CvCapture *capture;
    IplImage  *frame;
    double    t, ms = 0;

    bool finished = false;
    /* initialize webcam */
    capture = cvCaptureFromCAM(0);

    CvFont myFont;

    //Initialise font
    cvInitFont(&myFont,CV_FONT_HERSHEY_COMPLEX_SMALL ,1.5f,1.5f,0,1,8);

    while (!finished)
        /* display frame */
        frame = cvQueryFrame(capture);
        cvPutText( frame, "My Text", cvPoint(50,50), &myFont , cvScalar(128) );
        // use other functions to draw something 

    /* free memory */
    return 0;

It also depends if you want to write your own code, or if you want a simple off the shelf solution. If you are writing your own code, then have a look at some of my scene rendering components, https://sourceforge.net/p/visionmagic/code/ci/master/tree/src/modules/sceneRendering/ , not perfect, but will give you an idea on how to render things in OpenGL, and Qt.


There are no ready-made open-source solutions that incorporate both streaming video & sensor data from an Arduino that I am aware of; however, it wouldn't be too hard to build your own. You would need a webcam + software for live streaming, then you'd need to interface your Rpi with the Arduino in order to access sensors that are connected to the Arduino via Python (or other language). The easiest way of doing that is through the i2c bus, which will leave both USB ports on the Pi free, so you can connect a webcam, and even a USB wifi stick.

  • mjpg-streamer - if your webcam supports MJPEG, it is the best and least computationally expensive way of streaming live video from your Raspberry Pi.
  • RPi + Arduino via i2c - Tutorial for connecting the Rpi to the Arudino via i2c (uses Python SMBus library). You will need to modify the Arduino code to send the sensor data you need based upon the byte value sent from the Pi.

You would then edit the php page hosted by the mjpg daemon to add your sensor data that you would collect via Python and the SMBus library.

You can launch mjpg-streamer using the following:

./mjpg_streamer -i "./input_uvc.so" -o "./output_http.so -w ./www"

or for webcams w/o MJPEG support:

./mjpg_streamer -i "./input_uvc.so" -o "./output_http.so -y -w ./www"

You can then view the livestream at localhost:8080


I have done something similar using the Robot Operating System (ROS). It makes streaming data really easy, although it might be more than you're asking for, and I'm not sure how much support for RPi there is. Basically, after installing ROS, you would just need to download the mjpeg_server (http://ros.org/wiki/mjpeg_server) package for streaming video and rosbridge (http://www.ros.org/wiki/rosbridge_suite) for streaming other types of data. ROS probably already has support for your webcam, but you'd probably have to write ROS nodes to collect other sensor data and publish them to topics. With rosbridge, you can easily expose those topics to the web on websockets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.