life. It represents a new paradigm of
group-computer relationships, potentially transforming the way humans work and live. It also promises
opportunity for innovation in science
and technology that will enable humans and computers to collaborate
in new ways to accomplish things neither could have done on their own.
The research explored here is supported by the Cognitive and Immersive
Systems Lab, a research collaboration
between IBM Research and Rensselaer
[ 1] Kelly, J. Computing, Cognition, and the Future
of Knowing. IBM Research, 2015; https://www.
[ 2] Inside Rensselaer. Immersive environments in
action, Aug. 31, 2016; http:// www.insiderensselaer.
[ 3] Martialay, M. IBM Research and Rensselaer
Polytechnic Institute pioneer next-generation
cognitive environments for business decision
making. RPI Ne ws (Nov. 18, 2015); https://news.
[ 4] IBM Social Media. Ho w Cognitive Computing Can
Reshape the Enterprise. You Tube.com, 2015;
[ 5] Martialay, M. At the edge of a cognitive space.
RPI News (Dec. 12, 2016); https://ne ws.rpi.edu/
[ 6] Su, H., Goebel, J., and Kolb, J. IBM and RPI Join
Forces to Create People-Friendly Computers. IBM
Social Media, You Tube.com, Nov. 17, 2015; https://
[ 7] Battad, Z. and Si, M. Using multiple storylines
for presenting large information networks. In
Proceedings of the International Conference on
Intelligent Virtual Agents, 2016.
[ 8] Sharma, G., Braasch, J., and Radke R. Interactions
in a human-scale immersive environment: The
CRAIVE-Lab. In Proceedings of the ACM International
Conference on Interactive Surfaces and Spaces
(Niagara Falls, Canada, Nov. 6–9, 2016).
[ 9] Si, M., Battad, Z., and Carlson, C. Presenting large
data with anchor points. In Proceedings of the
International Conference on Interactive Digital
Storytelling (Los Angeles, CA, Nov. 15–18, 2016).
[ 10] Sen, A., Bringsjord, S., Radke, R., and Peveler, M.
To ward the cognitive classroom: Mathematical
physics. Immersive Education Summit (Padova, Italy,
June 21-23, 2016).
Hui Su is director of the Cognitive and Immersive Systems
Lab, a collaboration of IBM Research and Rensselaer
Polytechnic Institute in Troy, N Y. He was director of IBM
Cambridge Research Lab in Cambridge, MA, leading
research in cognitive user experience, and the Center
for Innovation in Visual Analytics and Center for Social
Business in Cambridge, MA. He is an expert in human-computer interaction, cloud computing, visual analytics,
and neural network algorithms for image recognition.
Copyright held by the Author.
Publication rights licensed to ACM. $15.00
erences. This step enables the room’s
“mouth” (to deliver audio feedback)
and “hands” (to deliver visual output
Research in creative interactive
multimodal storytelling will help create multimodal narratives to translate insights and discovery from large
amounts of data into advanced visualization and sonification in an immersive environment and allow humans
to further explore the information
through human-in-the-loop analysis.
The situations room would be helpful
in a range of situations. For example,
in a corporation with an aggressive
acquisition strategy, its mergers-and-acquisitions team consisting of, say,
Matt, the leader, and team members
Gordon, Jeff, and Hui might gather in
its situations room to try to identify the
right candidate company.
Matt will ask the room to suggest
companies worth considering as acquisition targets. In a flash, a wall-size screen will fill with companies
represented by circles of varying sizes
based on company revenue and by colors based on company profit margins.
Matt asks the room to show the team
companies that are publicly traded,
with revenue of more than $25 mil-
lion. A smaller set of companies then
appears on the screen. Hui approach-
es the screen, points to a circle and
says, “I would like to know more about
this company, ABC.” The room grabs
more information about the company
and visualizes it on another wall-size
screen. Gordon walks closer to the
second screen and swipes his hands
apart to zoom into the information
being displayed. When the topic of
discussion shifts from one screen to
another but Jeff remains focused on
the previous point, the room recog-
nizes his focus and related changes in
the group’s activity, then asks, “What
are you thinking, Jeff?” Jeff might
then talk about his concerns related
to company ABC and remind everyone
to also think about company X YZ. The
room detects these different points
of view and finds more information
about company ABC. The room then
uses a third wall-size screen to visu-
alize the revenue history and news
about ABC over the past several quar-
ters and says, “Three of ABC’s main
customers have reported significant
revenue growth and increased pro-
duction volume.” The room further
explains the details of all the relevant
stories through multimodal narra-
tives. Over time, the room learns the
mental habits and biases of the partic-
ipants, using this knowledge to pres-
ent information in a way that is most
useful to each of them. At the end of
the meeting, the team and the room
identify six companies to explore for
the next meeting.
In that meeting, the room recognizes the team and asks, “What can I
do for you?” Matt says, “Please bring
me the last working session.” The
room retrieves the material for the
six companies and displays it on the
three large screens. The room asks,
“Would you like to put the companies’ information into a decision table to compare?” Matt says, “Yes, and
also give me a suggestion.” The room
then displays a decision table with
all the companies and their information and also highlights a company
it recommends removing. Following
several rounds of discussions, they
settle on a single candidate company
and prepare for a CEO meeting to
The future cognitive and immersive
situations room is a new type of integrated system that will augment
group intelligence in the context of
business, government, or personal
Over time, the
room learns the
and biases of the
this knowledge to
in a way that is
to each of them.