and topic detection and tracking. Examples of applications are opinion-mining tools and market research
In the field of artificial intelligence,
the Open Web Index could be used as a
basis for large-scale machine learning.
Likely applications in this area are machine translation, question-answering,
and conversational applications.
Last but not least, an Open Web Index would provide a rich data source
for researchers in many different
fields, ranging from computer science and computational linguistics
to computational social sciences and
It is clear this short list of ideas is far
from being complete and only serves illustrative purposes. It shows, however,
the huge potential of making Web data
open to all parties interested.
Some alternative solutions have been
proposed for fostering plurality in the
search engine market. The first and
probably most obvious solution is to
wait for commercial market players
to develop alternatives. However, as
we have seen in the last 15 years or so,
Bing has been the only search engine
capable of gaining considerable market share. Other search engines have
failed, have been acquired by larger
search companies, or have focused on
niche markets. All new search engine
providers face the problem of having
to build their own index, which is, as
has been described earlier, a very costly undertaking. Furthermore, what
would be gained if we had one or two,
even three more search engines on the
market? From my point of view, the
problem lies not in having a few more
search engines, but in providing real
The second line of argumentation
says Google should be forced to provide
fair and unbiased results. This is what
the European Commission’s competitive investigation against Google has
been all about. However, as ranking
results are always based on interpretations (and human assumptions inherent in the ranking algorithms), there
is no such thing as an unbiased result
set. Only a multitude of different algorithmic interpretations can help bring
about search plurality.
The third line of argumentation
calls for Google to open its index to
third parties. Then, it would be possible to build (search) applications on
top of Google’s index. However, the
control over the index—and over what
third parties would be able to get from
the index—would still lie in the hands
of a private company, the index would
still not be transparent, and there
would still be no influence on how the
index is composed.
The fourth, and already widely discussed solution, is building a publicly
funded search engine as an alternative
to the commercial enterprises. However, this again would only add one more
search engine to the market, instead of
The main idea I presented in this Viewpoint is to foster building search engines and other services needing Web
data on top of a public infrastructure
that is open to everyone. A multitude
of such services would foster plurality
not only on the search engine market
(with the result of having more than a
few search engines to choose from) but
even more importantly, a plurality with
regard to the results users get to see
when using search engines.
Search results as a basis for
knowledge acquisition in society
seem too important to be left solely
in the hands of a few commercial
enterprises. The Open Web Index
is comparable to other public services such as constructing roads and
railroad tracks, supporting public
broadcasting and, most notably,
building a library system. An Open
Web Index could be one of the main
building blocks of the library of the
An Open Web Index is a project
that cannot and should not be un-
Those that benefit
from the index
should have their say
in building it.
dertaken by a single company or institution. On the contrary, I envision
building such an index as a task of
society and for society, meaning we
should build the index involving all
actors and interest groups relevant
to society at large. Those that benefit
from the index should have their say
in building it.
A question that remains is funding.
As a considerable amount of money
is needed, I argue for public funding
not by a single state, but rather by a
larger entity such as the European
Union. This, however, does not mean
a governmental body should also be
the operator of the Open Web Index.
Rather, it should be run by an organization that is relatively free from
state intervention. One could think
of a foundation running it or a model
similar to public broadcasting. Whatever the mode of operation, as a project of and for society, funding should
be applied for the greater good.
1. Broder, A. A taxonomy of Web search. ACM SIGIR
Forum 36, 2 (2002), 3–10.
2. Epstein, R. and Robertson, R.E. The search engine
manipulation effect (SEME) and its possible impact
on the outcomes of elections. In Proceedings of the
National Academy of Sciences 112, 33 (2015), E4512–
3. European Commission. Antitrust: Commission fines
Google € 2. 42 billion for abusing dominance as search
engine by giving illegal advantage to own comparison
shopping service—Factsheet, 2017; https://bit.ly/
4. Goel, S. et al. Anatomy of the long tail: Ordinary
people with extraordinary tastes. In Proceedings
of the Third ACM International Conference on Web
Search and Data Mining, ACM (2010), 201–210.
5. Grimmelmann, J. Some skepticism about search
neutrality. The Next Digital Decade: Essays on the
Future of the Internet 31, (2010), 435–460.
6. Lewandowski, D. Is Google responsible for providing
fair and unbiased results? In M. Taddeo and L. Floridi,
Hrsg., The Responsibilities of Online Service Providers.
Springer, Berlin Heidelberg, 2017, 61–77.
7. Noble, S.U. Algorithms of Oppression: How Search
Engines Reinforce Racism. New York University Press,
N Y, USA, 2018.
8. Purcell, K., Brenner, J., and Raine, L. Search Engine
Use 2012. Washington, D. C., USA, 2012.
9. Schwartz, B. Google’s search knows about over 130
trillion pages. Search Engine Land, 2016; https://selnd.
10. Sterling, G. Data: Google monthly search volume
dwarfs rivals because of mobile advantage. Search
Engine Land, 2017.
11. Sullivan, D. Google now handles at least 2 trillion
searches per year. Search Engine Land, 2016; https://
12. Sullivan, L. Report: Companies will spend $65
billion on SEO in 2016. Media Post, 2016; https://bit.
13. White, R. W. and Horvitz, E. Cyberchondria. ACM
Transactions on Information Systems 27, 4 (2009),
Article No. 23.
Dirk Lewandowski ( email@example.com)
is Professor for Information Research and Information
Retrieval at the Hamburg University of Applied Sciences in
Copyright held by author.