Even the most ardent NUI advocates agree with Rubin. Oblong’s Underkoffler, for instance, says that writers will likely be well served by the keyboard for the foreseeable future, but that those who design ship hulls or airplane wings would be better served by three-dimensional NUIs. Moreover, he says it is vital not to graft a notion of a new interface design by simply extending two-dimensional GUI concepts onto the prototypes of three-dimensional applications. These applications will need computational capabilities along not just the flat x and y axes— an example might be a wall-sized but still two-dimensional application for designing the very three-dimensional ship’s hull he mentioned—but will also need to compute the depth of the z axis. Oblong’s g-speak platform, based on work Underkoffler pioneered in the 1990s at the Massachusetts Institute of Technology’s Media Lab, computes this spatial environment via networked computers and screens that allow rich three-dimensional interaction. Ultimately, Underkoffler thinks a hybrid UI ecosystem will evolve.

“We’re not out to replace the keyboard; let it do what it’s best at,” he says, “but when it comes to designing airplane wings, you do need a spatial UI. So, it’s about situating the right activities in the right interaction zone.”

 

Homebrewed algorithms

Since the Microsoft SDK precludes commercial use, many early academic and enterprise projects using PrimeSense and/or stripped down Kinect hardware use either homebrewed algorithms or open source drivers and middleware released by consortia such as OpenCV or OpenNI, the natural interface forge formed in November 2010, by PrimeSense and robotics pioneer Willow Garage. OpenNI leverages the PrimeSense depth-sensing technology, which is processed in parallel by PrimeSense’s system-on-a-chip processor after receiving coded near-infrared light from its partnered CMOS sensor.

OpenNI supplies a set of APIs to be implemented by the sensor devices, and a set of APIs to be implemented by the middleware components. Thus, OpenNI’s API enables applications to be written and ported to operate on

“People don’t buy
motion control,”
amir Rubin says.
“they buy the
experience being
delivered to them.
the best input
device is something
you don’t remember
is on you.”

top of different middleware modules. It also enables middleware developers to write algorithms on top of raw data formats, regardless of which sensor device has produced them, and offers sensor manufacturers the capability to build sensors that power any OpenNI-compliant application.

In fact, one nascent healthcare industry application partially built on open source stacks by a team of surgeons and engineers at Sunnybrook Health Sciences Center in Toronto for the Kinect camera is already drawing attention.

Allowing surgeons access to medical images while not having to touch a controller—and thereby saving them the necessity to re-scrub in order to preserve sterility around the patient— is an early enterprise triumph for the NUI concept. Computer vision specialist Jamie Tremaine says the gesture-based UI he and his colleagues developed has proven exceptionally robust and enables surgeons to view through MRI and CT scan samples that can run from 4,000 to 10,000 slides without ever having to re-scrub.

For such an application, the hand
and arm gestures recognized by the
Kinect camera are suitable, but Tre-
maine says “a lot of the work we’ve
done hasn’t even been on the techni-
cal side as much as creating gestures
in the operating room that allow very
fine-grained control, but which have to
be larger.”
Another NUI developer, Evan Lang
of Seattle-based UI design firm Iden-
tityMine, says his work with trying to
develop Kinect NUIs (on PrimeSense
drivers) similar to current GUI com-
mands revealed vexing user issues. In
developing a Web button, for instance,
Lang says, “I programmed it to rec-
ognize a poking gesture, where you
move your hand quickly forward and
quickly back. When I got some test
users to try it out, and said ‘Poke it or
press it,’ everybody had a very differ-
ent idea of what that actually meant.
Some did a kind of poking thing. Other
people moved their hand forward but
wouldn’t move it back, and others,
who were very cautious and deliberate
about it, the machine wouldn’t regis-
ter as a poke.”
Oblong’s Underkoffler says prob-
lems such as Lang encountered are
emblematic of grafting current GUI-
based mechanics on an idea that needs
something else.

output Perceptions

Robotics researchers such as Nicholas Roy, associate professor of aeronau-