Since BrainBoost ( www.brainboost.
com) is a commercial system, little is
known outside the company about the
algorithms it employs. Nevertheless,
it quickly gained popularity among
bloggers and other online information
seekers, since it delivers decent accuracy and quick response. Answers.
com bought BrainBoost for $4 million
in cash and shares of restricted BrainBoost stock in 2005.
Another prototype developed by Language Computer Corporation ( www.lan-guagecomputer.com) returns up to 10
answer snippets, with the words from
the question (not the precise answer
itself) highlighted. AnswerBus (
miss-hoover.si.umich.edu/~zzheng/qa-new/)
table 2: comparing search-engine performance: Google and msn (as a team)
vs. selected online Qa systems (as a team).
Question
Aspartame is also called what?
At what speed does the earth revolve
around the sun?
At what time of year is air travel at a peak?
boxing day is celebrated on what date?
Cnn is owned by whom?
how big is our galaxy in diameter?
how did Al Capone die?
how did bob marley die?
how far is it from denver to Aspen?
how far is pluto from the sun?
how long can a british prime minister serve in
office?
how many copies of an album must be
sold for it to be a gold album?
how many stradivarius violins were
ever made?
how many teachers are there in the u.s.?
how much folic acid should an expectant
mother get daily?
in what country is a stuck-out tongue
a friendly greeting?
What color is a giraffe’s tongue?
What continent is Argentina on?
What continent is italy on?
What do you call a professional map drawer?
What famous model was married
to billy Joel?
What is the collective noun for geese?
What is the collective term for geese?
What is the islamic counterpart to
the red Cross?
What is the largest city in Wisconsin?
What is the largest snake in the world?
What is the largest variety of cactus?
What is the most heavily caffeinated
soft drink?
What ocean did the titanic sink in?
average score across
all questions:
average mRR average
Google msn for the search mRR for the
mRR mRR Portals team Qa team
0.00 0.33 0.17 0.29
0.00 0.50 0.25 0.25
and NSIR ( tangra.si.umich.edu/clair/
NSIR/html/ nsir.cgi) were the two earliest open-domain Web QA systems developed in academic institutions, and
their algorithms are detailed in a number of publications. 5 Based on matching
the question to a trained set of answer
patterns, ASU QA uses probabilistic triangulation to capitalize on the redundancy of publicly available information
on the Web. Along with BrainBoost,
ASU QA was used for several years in a
$2 million project supported by NASA
( www.aee.odu.edu) aimed at developing collaborative distributed engineering knowledge/information management systems and intelligent synthesis
environments for future aerospace and
other engineering systems.
0.00 0.00
0.50 1.00
0.20 0.50
0.14 0.00
0.00 0.00
0.17 0.00
0.11 0.00
1.00 0.11
0.00 0.00
0.00
0.75
0.35
0.07
0.00
0.08
0.06
0.56
0.00
0.00
0.63
0.65
0.65
0.21
0.51
0.29
0.75
0.13
1.00 0.00
0.50
0.35
0.00 0.00
0.00
0.38
0.00 0.00
0.20 0.11
0.00
0.16
0.08
0.25
0.25 0.00
0.13
0.00
0.50 1.00
0.33 0.00
0.25 0.00
0.00 0.00
1.00 0.13
0.75
0.17
0.13
0.00
0.56
0.63
0.63
0.41
0.00
0.07
0.17 0.33
0.33 0.20
1.00 1.00
0.25
0.27
1.00
0.75
0.81
0.68
0.33 1.00
1.00 0.50
0.00 0.00
0.00 0.00
0.67
0.75
0.00
0.00
1.00
0.81
0.33
0.05
0.20 0.14
0.30 0.24
0.17
0.27
0.68
0.42
Beyond Keywords
Comparing and evaluating different
Web QA systems is not straightforward
and, to our knowledge, has never been
done before the study we describe here.
In the annual TREC competition, the
rules are set in advance, and participating researchers approximately predict
the distribution and types of questions
that would be expected from their experience in prior years. Meanwhile,
the objectives of each Web QA system
are different. The commercial systems
(such as Ask.com and BrainBoost) are
primarily interested in increasing traffic volume and visibility online to generate maximum potential advertising
revenue or investment capital. The research prototypes (such as ASU QA and
NSIR) are primarily interested in demonstrating innovative ideas in certain
unexplored fields of research involving
information seeking, not in competing
with commercial systems. As a result,
the systems we consider here support
different sets of features and interfaces,
as in Table 1.
The goal of our study in spring 2005
and repeated in 2007/2008 was not to
compare QA systems against each other
but to determine whether any of them
might offer additional power relative
to keyword search engines, exemplified by Google and MSN. In particular,
we wanted to know whether automated
QA technology provides answers to certain questions that keywords may find
difficult or impossible to answer. For
this reason, we performed an informal
comparison of the QA systems in Table
