Overview:
The goal of this prototype was to demonstrate how a ball’s movement around an inclined surface can be the input for an interactive experience.
Using a Kinect depth sensor and OpenCV blob tracking, you can track a ball’s movement around a flat surface. For our purposes, we were running the Kinect and the Cinder Ball Tracking App on a Windows computer running a Spacebrew server. The only data being sent through the server is the the blob’s centroid XY coordinates. This guide will focus less on the actual ball tracking and more on how to use the output to trigger different elements.
Ver. 1: Smart LED Strips
For the first Ball Tracking prototype, we wanted to have the ball’s movement along the Y-Axis of the inclined surface to effect a pair of Smart LED strips. We used the Adafruit NeoPixel Digital RGB LED Weatherproof Strip 60 LED for our LED Strips. These strips come with embedded control chips making it possible to access each LED individually for full control of color and brightness.
Step 1: SpaceBrew->Cinder->Serial
The Ball Tracking app sends XY coordinates from the blob centroid through SpaceBrew. First, you’ll need a Cinder App that can connect to the SpaceBrew server to receive the XY coordinates and send them out as simple serial data. After a simple serial handshake to insure the Arduino is listening for incoming Serial, you’ll write the incoming SpaceBrew message into a string and send it to the serial port.
Step 2: Serial-> Arduino-> NeoPixel Strips
Next, you will hook up the LED strip to power, ground and the signal wire to the the Arduino analog pin. The strip will also need to share an additional ground with the Arduino. Download and include the Arduino library, Adafruit_NeoPixel. Adafruit has a great tutorial on setting this up, and more info can be found here. This library allows simple functions to access each pixel individually. Finally, scale the incoming Y-coordinate to the length of of the LED strip and send it to a function to set each LED color and add some color trails that fade out as the ball travels up and down the surface.
Ver. 2: Projected Graphics
For the second Ball Tracking prototype, we wanted to have the ball’s movement affect the graphics being projected onto the surface. We used an already existing modular graphics app that can take in spatial data and fire different graphic elements.
SpaceBrew-> Cinder Graphics App-> Projector
Like the version above, this app receives the same XY coordinates from SpaceBrew. This time you will take XY coordinates and map them to a canvas to trigger the graphics elements. The only problem areas for this was finding the correct alignment between the blobs centroid and the projection. Any misalignment or lagging is easily noticeable and ruins the illusion of the graphics emerging from the ball’s path.
Hardware
- Arduino Mega
- Adafruit NeoPixel Digital RGB LED Weatherproof Strip 60 LED (http://www.adafruit.com/products/1138)
- 5V AC power supply for LED Strips
- Kinect for Windows
- Windows 8 PC for Cinder-Kinect-App using the MS Kinect SDK
- Mac Mini for Graphics-App
- Projector
- Ball
Final Thoughts
Using OpenCV blob tracking to track a ball around a flat surface proved to be an effective way to trigger both the smart LED strips and the projected graphics. The obvious shortcomings would be trying to track a ball on a surface with any sort of obtrusions or obstacles as well as tracking multiple balls. Because it is only looking for blobs at the depth right above the surface, it is easy to get the the only blob we are looking for. If anything else was sticking up in that same threshold, we would have to find a way to differentiate between our ball and any other blob. One solution to this would be to use either frame differencing or a combination of frame differencing and blob tracking.