Your Wii remote has probably been gathering dust in a drawer or down the back of a sofa somewhere. Well, it’s time to dig it back out because researcher Johnny Lee’s 2008 TED talk “Wii Remote Hacks” came onto my radar this week and it turns out there’s plenty of life left in the old white brick.
There’s plenty more where that came from, too. Across Johnny Lee’s YouTube channel there’s a wealth of hacks and workarounds that can turn the cheap video game controller into any number of otherwise expensive pieces of hardware.
In one of Lee’s first videos, he demonstrates his use of the Wii remote to create foldable displays. The controller is responsible for tracking the display objects and allowing display projections to react in real time to physical manipulations of those objects. The results are pretty cool. His opening and closing of umbrella and fan displays are impressive, but it’s when he starts flipping objects to change the document view – turning it the other way round for double sided documents or changing the display depending on the angle of viewing – that things get really interesting.
Lee uses an example application of his workaround that i’m surprised hasn’t picked up more traction in the years following his initial research. By tracking the angle of the display object and altering the display projection accordingly, Lee is able to project different documents depending on who is looking at them. For example, when a display object is placed face up on a table it will show a public view but when angled toward the viewer, a private view is displayed.
Tracking Fingers With The Wii Remote
While it’s difficult to see why the idea of different displays based on angle hasn’t caught on, it’s pretty easy to understand why Lee’s finger tracking is currently without commercial development.
The idea, nevertheless, remains sufficiently cool.
The Wii remote infrared camera is able to pick up reflected sources of infrared light. So, using an infrared LED array with a hole in the middle to poke the controller through, Lee’s fingers reflect small portions of IR light back to the controller and become trackable. We quickly see, however, that there’s a lot of noise around the specific points Lee wants to track, and moving too far from the light source quickly diminishes the effect.
Strapping reflective tape to each finger quickly remedies this, and allows for clean tracking from further distances. The Wii remote is capable of tracking up to four points, so multi-touch is smooth and easy!
Those four tracking points shine particularly brightly in Lee’s interactive whiteboard application. It gained the biggest applause of all his demos at his TED talk and has already brought the interactive whiteboard to thousands of classrooms that would have otherwise be unable to afford such technology in 2008. It’s particularly impressive when you remember that interactive whiteboards had only been around a few years before Lee hacked his way into the space. I was 12 at the time of Lee’s TED talk and can remember their installation in my primary school a few years before, yet here was a brand new way of achieving such solutions for a fraction of the price.
The pen used is a simple biro kitted out with a battery and an infrared light tip, and a Wii remote is positioned off to the side of the display. Lee notes in his YouTube demonstration that it’s important to find a sweet spot where the remote can still see the entire display with its 45 degrees of vision while not being too far back to track smoothly. With this tech, any surface you can project onto can be turned into an interactive whiteboard or tablet display with multi-touch capabilities.
Head Tracking for Desktop Virtual Reality Displays
For a 3D desktop virtual reality display to work, the computer need to know the location of your head in relation to the screen to know what to show you and where – this is head tracking. For this, Lee uses both the Wii remote and the Sensor Bar but in reverse order, positioning the remote underneath the computer screen and moving the Sensor Bar around in front of it.
The Sensor Bar is, essentially, two sources of infrared light. So to avoid having to strap a plastic brick to your forehead, Lee encourages users to purchase safety glasses that often come kitted out with LES lights on the side. These LED bulbs can be replaced with IR ones to reproduce those emitted by the Sensor Bar.
The results are amazing, with 3D targets popping out of the screen and dynamically adjusting to precise positions of your head. Pointing and shooting targets with the Wii remote makes for an incredibly impressive gameplay experience, and Lee even implores developers to consider the potential of this application in their creations.
The sophistication of these applications may be watered down in 2018. We’ve seen incredible advances in technology in the past 10 years and life changing applications of that technology to boot. However, taking the time to look back on what we could already achieve with household items back in 2008 may just inspire others to take another look at what they have lying around today. You can bet there’s significantly advanced tech waiting in a drawer or cupboard and fishing it out may just lead to some equally inspiring creations!
You can read about all of these applications in full in the Lee’s paper “Hacking The Nintendo Wii Remote”