Microsoft Has A Mindblowing Display Technology, But Doesn't Want To Talk About It

In September, Microsoft filed a patent application for what was essentially a holodeck. Now, that technology is a step further to becoming reality.

At the Consumer Electronics Show, people got a glimpse of a product Microsoft calls "Illumiroom," which the company describes "peripheral projected illusions for interactive experiences." 

Microsoft Research posted a video demo of the technology on YouTube a week ago, and in that time the video has received more than 3.6 million views.

And for good reason. This is mindblowing technology. But now, for some reason, Microsoft doesn't want to talk about it.

Essentially Illumiroom is a projector that works in conjunction with a conventional television to project some of the content on the edges of the screen onto the walls, floors and ceiling. Illumiroom creates the illlusion that a game played on a Microsoft Xbox, for example, overflows the boundaries of the television screen.

Illumiroom was designed by Brett Jones, Hrvoje Benko, Eyal Ofek and Andy Wilson, all Microsoft researchers who focus on natural interaction and/or augmented reality. Benko, for example, helped turn a research project on gesture interaction with surfaces into 2010's Microsoft Touch Mouse, whose surface can be stroked and manipulated, a gesture-based peripheral launched two years before Windows 8 ushered in touch screens into general use.

The video of Illumiroom was included in a CES keynote given by Samsung. But the technology did not make an appearance elsewhere at the show. (For the first time in many years, Microsoft did not have a booth at CES.)

Microsoft representatives said they would not comment further, and declined to allow the researchers to speak to the press.

A New Spin On An Old Idea

Peripheral illumination has existed for some time. Philips launched the Ambilight technology in 2002, which mounted rear- and side-mounted LED lights to the top and sides of a television. The effect smeared the color scheme that the television displayed on the walls behind the display, apparently in an attempt to reduce eye strain. 

Illumiroom is basically that idea but on steroids. According to the video demonstration of the technology (see below), actual scenes from the game are projected outwards, such as "flames" that appear on the screen and what looks to be some sort of a lightning gun.

Illumiroom apparently adds to the screen, showing objects that otherwise would not be displayed, "extending" elements on the screen onto other surfaces in the room. It's unclear whether Illumiroom does that automatically or whether the game developer would need to code those extensions in via an API of some sort.

You'll also note that the projector doesn't appear to to draw the peripheral scene in as much detail as what's shown on the main screen. Eventually, the graphics hardware is going to run out of available horsepower, assuming that the Xbox is controlling both the console's video output as well as the projector. 

How Does It Work?

So far, Microsoft's demonstration gives no indication that Kinect is being used to evaluate and respond to the player's gestures, as the original patent application indicates. An improved Kinect-like device could also be used to track the player’s eyes or face, letting those serve as controls, the patent suggested.

At least according to the video, the Illumiroom technology works quite well. The shooter demonstration is particularly impressive, with the widened perspective considerably altering the game experience. Still, objects in the periphery are somewhat vague, creating a clear differentiation between the "focused" scene on the television and the muddier peripheral scene. And the starfield, projected onto the walls and floor, creates an unfortunate "mirror ball" effect.

As we noted before, several questions remain unanswered, including the number of projectors being used, how and where they would need to be mounted, and whether a cheap Kinect-like peripheral could do double duty as both a projector and movement sensor.

Still, any questions may have to wait to be answered at the end of April, when Jones and his colleagues present a paper describing Illumiroom at the ACM SIGCHI Conference on Human Factors in Computing Systems, the premier international conference on human-computer interaction, in Paris. 

Ambilight photo by Stephan Lebachev.