For Liberty Science Center’s refreshed Energy Quest exhibition, Red Paper Heart created a new interactive installation about a simple but powerful idea: energy is all around us.
If we could easily and reliably collect the energy in sunlight, wind, water, heat, and matter, our energy problems would look very different. The installation turns that idea into a physical interaction. Visitors collect different forms of energy in small cube-shaped cups, carry them across the space, and pour them into a shared touchscreen table.
It sounds simple from the outside. Pick up a cup. Fill it with energy. Pour it into the table.
But for that illusion to work, the system has to understand a physical object moving through the real world. It has to know which cup was filled, what kind of energy it contains, where it is above the table, and when it is being turned over to pour.
This article is about how we built that tracking system.
The Interaction
The installation is built around five LED paths that travel from one side of the wall, across the ceiling, and down the other side. Each path represents a different form of energy: hydro, geothermal, wind, atomic, and solar.
Each path has a filling station. Visitors place a cup on a small shelf, the cup fills with that station’s energy type, and the visitor brings it to the table. When they turn the cup over, the energy pours into the digital world on the touchscreen.
The challenge was not just to make the table react. The challenge was to make the physical gesture feel believable.
Two Different Tracking Problems
The interaction has two very different sensing problems.
At the filling stations, the cup sits in a predictable place. We only need to know which cup is there and which energy type it should receive. That is exactly the kind of problem RFID is good at. We have used RFID in many projects before, and for this part of the installation it was the obvious choice.
Pouring into the table was different.
The cup is moving. It can be held at different heights. It can rotate in any direction. Visitors might interact with it in unpredictable ways, or approach the table from different angles. And the response on the screen needs to appear in the right place, close enough to the physical cup that the illusion holds.
That was the real puzzle.
Museum Constraints
The constraints shaped the technology as much as the concept did.
Each cup needed to be inexpensive. We did not want expensive electronics inside an object that could be dropped, lost, stolen, or damaged. The cups needed to be easy to replace and rugged enough for heavy daily use by children and families.
The system also needed to be maintainable. This is a museum installation, not a short event or a weekend prototype. It should run reliably for years. That ruled out anything with batteries in every cup, complicated pairing procedures, fragile embedded hardware, or large recurring licensing costs.
We also did not know the final form or material of the cup during early technical development. The tracking approach needed to survive design changes. Ideally, it would work with almost any cup, without requiring major changes to the sensing system.
The Technologies We Considered
We looked at several possible approaches.
RFID was already part of the installation, and it was excellent for detecting a cup at a filling station. But RFID is not a natural fit for tracking the position and rotation of a moving cup in 3D space, especially while someone is turning it upside down over a table.
Bluetooth beacons or other wireless positioning systems were possible, but they would have introduced electronics into every cup. That meant batteries, charging, cost, failure points, and maintenance. For a museum installation designed to run for many years, that was not attractive.
We also looked at camera-based detection, including object detection models such as YOLO. These systems can be powerful, but they introduce other questions: training data, lighting conditions, occlusion, long-term maintenance, the complexity of reliably estimating the precise position and rotation of an object, and in some cases the licensing or subscription costs associated with well-trained commercial models.
The approach that became most promising was simple marker tracking: placing a visual marker on the cup and using a camera to detect it.
Specifically, we started testing ArUco markers with OpenCV.
Prototype One: ArUco Markers and a Regular Camera
Our first prototype used a regular camera, Python, OpenCV, and ArUco marker detection.
The early results were encouraging. OpenCV could detect the 2D corners of the marker very quickly. Once the marker was visible, the system knew exactly which marker it was looking at and where its corners were in the image.
In theory, we could estimate the 3D position of the marker using the camera intrinsics. In practice, even after calibration, the estimated 3D position was not accurate or stable enough for the kind of interaction we needed.
We needed the digital pour to appear where the cup physically was. If the tracking drifted or jumped, the illusion would break immediately.
So we extended the approach.
Prototype Two: Adding Depth
The next idea was to combine ArUco tracking with a depth camera.
Instead of using a camera image alone to estimate 3D position, we used the color image to detect the ArUco marker and locate its corners, then used the camera’s synchronized depth data to convert those 2D corner positions into 3D points.
From those points, calculate the position and rotation of the cup.
We built this prototype in TouchDesigner because it made it very fast to get the camera running and visualize the result. The camera we used was the Orbbec Femto Mega, one of the successors to Microsoft’s Azure Kinect.
This prototype proved that the system could work.
Once the marker was visible, the tracking was solid. We could get accurate position and rotation data. We could see when the cup was being picked up, when it was turned over, and where it was pointing toward the table.
That was the moment the interaction started to feel possible.
The Tradeoffs
No tracking system is perfect. This one had its own limitations.
The most obvious limitation is that the camera needs to see the marker. If a visitor covers the marker with a hand or finger, the system cannot detect it. That is the nature of vision-based tracking.
The second limitation is about the pouring gesture itself. The marker becomes visible only once the cup is turned over enough for the camera to see it. In our tests, that meant the cup had to rotate roughly past 110 degrees before tracking became reliable. In real life, liquid would begin pouring much earlier.
But the installation is a simulation. We are not measuring actual liquid. We are creating a readable, playful gesture that makes sense in the context of the experience.
The Advantages
The advantages were significant.
The expensive and fragile parts of the system — the camera, computer, and processing — are out of reach of visitors. The cup itself can remain simple and durable. There are no batteries in the cup. No charging. No electronics to fail. No wireless hardware to maintain.
If a cup breaks or disappears, it can be replaced. As long as the replacement has the right marker and RFID tag, it becomes part of the system.
The tracking is also surprisingly robust. A visitor can shake the cup, drop it, turn it around, or handle it roughly. As soon as the marker is visible again, the system immediately knows where the cup is and how it is oriented.
A nice bonus was that the same marker-based system could also help us calibrate the table. Because the camera can detect known markers in 3D space, we can use that data to align the virtual and physical coordinate systems more easily.
That kind of practical advantage matters a lot in permanent installations.
From Prototype to Production
The TouchDesigner prototype was extremely useful. It lets us move quickly, test the idea, and prove the core interaction.
But it was not the final production system.
The prototype still had performance limitations. It also still lacked many of the features we needed for production: configuration tools, calibration workflows, more debugging views.
So we moved the tracking system into a C++ application built with Cinder.
This was also the first project where we used AI-assisted development from beginning to end in a serious way. The AI did not magically “build the system,” but it became incredibly useful as a development partner: helping translate the TouchDesigner prototype into C++, explaining unfamiliar APIs, and speeding up the process of adding features.
The development still required close direction, testing, correction, and plenty of human judgment. But as a tool for moving between prototype and production, it was genuinely helpful.
The final system runs quickly, even on less powerful computers. We also heavily used ImGui to build internal controls and debugging tools, which became essential for tuning, testing, and maintaining the system.
The Final System
A working prototype proves that an idea can work. A public installation has to work every day.
In the final installation, the cup is intentionally simple. It does not need batteries, wireless radios, processors, or charging contacts. It carries an RFID tag so the filling stations can identify it, and a visual marker so the overhead camera can track it when it is poured.
That separation became the key. The visitor handles something durable and low-tech. The installation handles the complexity invisibly.
For the final system, we built production software with calibration tools, configuration controls, diagnostics, and the performance needed for daily operation. Visitors never see that layer, but it is what makes the visible experience reliable.
That is often where interactive installation work succeeds or fails. The magic depends on practical details: alignment, calibration, recovery behavior, replacement procedures, and the ability to understand what is happening when something does not behave as expected.
Technology Curation
A lot of interactive installation work lives in the space between invention and practicality.
There is a temptation to think that the most innovative solution must be the newest, most complex, or most technically impressive one. But in a permanent museum environment, novelty is only useful if it survives contact with real visitors.
For this project, the breakthrough was not a single piece of technology. It was the combination: a passive RFID tag, a printed marker, an open-source computer vision library, a depth camera, a custom C++ application, and a physical interaction simple enough for a child to understand immediately.
None of those pieces are magic on their own. Together, they make the cup feel like it is carrying energy.
In the end, the best tracking system is not the one visitors notice. It is the one that disappears.
They pick up a cup.
They fill it with energy.
They pour it onto the table.
And for a moment, invisible energy becomes something they can hold.
Leave a Reply