lion, and Rutter, whose total winnings amounted to $3.25 million, the most money ever won by a single “ Jeopardy!” player. At the end of the three-day event, Watson finished with $77,147, beating Jennings, who had $24,000, and Rutter, who had $21,600. The million-dollar prize money awarded to Watson went to charity.

Named after IBM founder Thomas
J. Watson, the Watson system was built
by a team of IBM scientists whose goal
was to create a standalone platform
that could rival a human’s ability to
answer questions posed in natural
language. During the “Jeopardy!” chal-
lenge, Watson was not connected to the
Internet or any external data sources.
Instead, Watson operated as an inde-
pendent system contained in several
large floor units housing 90 IBM Power
750 servers with a total of 2,880 pro-
cessing cores and 15 terabytes of mem-
ory. Watson’s technology, developed by
IBM and several contributing universi-
ties, was guided by principles described
in the Open Advancement of Question-
Answering (OAQA) framework, which is
still operating today and facilitating on-
going input from outside institutions.
Judging by the sizeable coverage of
the event, Watson piqued the interest
of technology enthusiasts and the gen-
eral public alike, earning “Jeopardy!”
the highest viewer numbers it had
achieved in several years and leading
to analysts and other industry observ-
ers speculating about whether Watson
represents a fundamental new idea
in computer science or merely a solid

feat of engineering. Richard Doherty,
the research director at Envisioneering
Group, a technology consulting firm
based in Seaford, NY, was quoted in an
Associated Press story as saying that
Watson is “the most significant break-
through of this century.”
Doherty was not alone in making
such claims, although the research-
ers on the IBM team responsible for
designing Watson have been far more
modest in their assessment of the
technology they created. “Watson is a
novel approach and a powerful archi-
tecture,” says David Ferrucci, director
of the IBM DeepQA research team that
created Watson. Ferrucci does charac-
terize Watson as a breakthrough in ar-
tificial intelligence, but he is careful to
qualify this assertion by saying that the
breakthrough is in the development of
artificial-intelligence systems.

“The breakthrough is how we pulled everything together, how we integrated natural language processing, information retrieval, knowledge representation, machine learning, and a general reasoning paradigm,” says Ferrucci. “I think this represents a breakthrough. We would have failed had we not invested in a rigorous scientific method and systems engineering. Both were needed to succeed.”

 

Contextual Evidence The DeepQA team was inspired by several overarching design principles, with the core idea being that no single algorithm or formula would accurately understand or answer all questions,

 

Watson’s on-stage persona simulates the system’s processing activity and relative answer confidence through moving lines and colors. Watson is shown here in a practice match with Ken Jennings, left, and Brad Rutter at IBM’s Watson Research Center in January.

says Ferrucci. Rather, the idea was to build Watson’s intelligence from a broad collection of algorithms that would probabilistically and imperfectly interpret language and score evidence from different perspectives. Watson’s candidate answers, those answers in which Watson has the most confidence, are produced from hundreds of parallel hypotheses collected and scored from contextual evidence.

Ferrucci says this approach re-
quired innovation at the systems
level so individual algorithms could
be developed independently, then
evaluated for their contribution to the
system’s overall performance. The ap-
proach allowed for loosely coupled in-
teraction between algorithm compo-
nents, which Ferrucci says ultimately
reduced the need for team-wide agree-
ment. “If every algorithm developer
had to agree with every other or reach
some sort of consensus, progress
would have been slowed,” he says.
“The key was to let different mem-
bers of the team develop diverse algo-
rithms independently, but regularly
perform rigorous integration testing
to evaluate relative impact in the con-
text of the whole system.”
Ferrucci and the DeepQA team are
expected to release more details later
this year in a series of papers that will
outline how they dealt with specific as-
pects of the Watson design. For now,
only bits and pieces of the complete
picture are being disclosed. Ferrucci
says that, looking ahead, his team’s re-
search agenda is to focus on how Wat-
son can understand, learn, and interact
more effectively. “Natural language un-
derstanding remains a tremendously
difficult challenge, and while Watson
demonstrated a powerful approach,
we have only scratched the surface,” he
says. “The challenge continues to be
about how you build systems to accu-
rately connect language to some repre-
sentation, so the system can automati-
cally learn from text and then reason to
discover evidence and answers.”
Lillian Lee, a professor in the com-
puter science department at Cornell
University, says the reactions about
Watson’s victory echo the reactions fol-
lowing Deep Blue’s 1997 victory over
chess champion Garry Kasparov, but
with several important differences.
Lee, whose research focus is natural

Photo courtesy IBM