Microsoft Goes To The Mall With Kinect Features For Wall-Sized Displays

Maybe touch will never be a particularly good match for Microsoft's Windows. Its Kinect sensor, however — yes, the one mostly used as an attachment for its Xbox 360 game system — might be another story.

Today, Microsoft announced two new gestures for Kinect for Windows, the PC-specific version of its Kinect interface. Within the new Kinect for Windows SDK 1.7 are two new gestures: the "push" or "click" gesture, and a closed-fist gesture for scrolling and panning a screen. Both could help power interactive wall-sized displays at your doctors' office — or the mall.

Not An Actual Product — Yet

Right now, Kinect for Windows is more of a research tool than an actual product. The sensor itself costs $228.99 at Amazon, is designed for closer sensing, and comes with a warning that it must be used as either a development tool or with software that has been designed for it - which, really, none has. But all that's changing.

"We actually think that this will be the biggest advancement [in Kinect for Windows] since the original introduction," said Bob Heddle, who leads the Kinect for Windows team in the Interactive Entertainment Business (IEB) Group at Microsoft. Microsoft calls the new gestures Kinect Interactions.

If the closed-fist gesture sounds familiar, that's because Microsoft demonstrated it at its recent TechFest, where Microsoft's vision for the future was embodied in wall-sized displays. In that environment, touch doesn't work as well as simply interpreting gestures via Kinect, which can "read" a user's gestures and translate them into a familiar user interface.

Windows On Walls

Microsoft sees those wall-sized displays, powered by Kinect for Windows, in three areas: interactive displays in public spaces, health care, and education and training. Some of Microsoft's vision is already a reality: at Bloomingdale's in New York, FaceCake and Microsoft installed "virtual dressing rooms" last year, where shoppers could use Kinect to "try on" a virtual scarf or dress, layered on top of their video "reflection". And there are virtual exercises of a sort, too, with "Just Dance" and similar Kinect-powered games grading players on how well they copy dance moves. 

The next step, Heddle said, was to make those activities more purposeful. Instead of dancing, for example, Kinect could be used as a rehab tool, working with a virtual physical trainer to ensure that users were putting in the required work. Microsoft also hopes to make Kinect an interface for "dirty hands" environments, such as using a PC when it might be covered with grease, for example. (Samsung's Galaxy S4 also allows users to swipe left and right, plus preview content, by simply moving their hand near the screen.)

"But to do that, we need a selection mechanism," Heddle said. "The value is that we can see you, understand the way you're looking, and draw the clothing on you... But to do that, you need to allow the user to choose the clothing, and now there's a 2D UI component."

How It Works

If you think about it, operating a PC-like environment doesn't have to be much different than, say, playing the giant electronic keyboard at FAO Schwarz, as Tom Hanks and a friend did in Big.

To gain control of the display, users raise their hand, identifying themselves to the screen. That hand appears as a wavy circle, a cursor that can be moved across the screen by moving your arm. Users "click" by pushing in toward the screen. (There is apparently no double-click in the Kinect interface.) What the new SDK adds is the closed-fist "scroll" gesture, which essentially works a swipe. Close your fist and move your hand down, and the window moves down. Chop your closed hand down rapidly, and the windows scrolls rapidly, like a flicked page on a smartphone.

According to Heddle, the new Kinect interface presents a familiar, Windows-like environment for interacting with large-screen content. Small-screen content, too. Last year, Microsoft Research quietly unveiled the Kinected Browser — a set of browser elements that let developers write what are essentially Kinect-aware Web apps. A future with Kinect-equipped PCs won't necessarily need dedicated software or apps.

Kinect Fusion

Microsoft also announced Kinect Fusion, essentially the opposite of a 3D printer. With Fusion, the Kinect sensor functions as a wand, which a user can move around a statue or other object, transforming it into a virtual object using voxels, or pixels that exist in a 3D space. While there's a limit to the number of voxels that can be stored, the detail of the 3D object can be improved by merely moving the Kinect sensor slowly around, allowing it to take more "pictures" of the object, Heddle said.

Essentially, Kinect Fusion would allow users to "scan" their own physical objects, then manipulate them and print them out. It's a different take on the same problem that Autodesk attempted to solve in 2011 with a process that could create 3D models from two-dimensional photos.

In a mall kiosk, a retailer could use Kinect to simply "scan in" a new piece of merchandise, Heddle said.

The Future Of User Interfaces Is... Microsoft?

It's all part of the what Microsoft sees as its future: natural user interfaces, driven by speech and touch and gestures. Touch might not work naturally on the PC, but it adds a degree of public performance to any retailer. Think about a wall-sized display projected onto the wall of a Microsoft store, or Williams-Sonoma, or JC Penney. Who wouldn't stop to watch someone waving at a giant LCD screen, and seeing it react?

There are still problems to solve. By raising one's hand, Kinect knows that a user is trying to use it. But that user has to relinquish control for another user to have a turn. In reality, the first user will probably walk away when they're done; however, other users might want a turn in the meantime, causing a fight for control. And even though Kinect for Windows was designed for Windows 8, it still can't control it, Heddle said.

Nevertheless, Kinect for Windows allows brands to engage with users, and users to engage with brands. A UI that some see as a hassle on the desktop becomes fun in the mall — and the only content to engage with is the brand that paid for it. You might call that a selling point.

Images: Microsoft, Amazon (Just Dance)