4. More broadly, what is NELL unable to learn, and what AI architecture is necessary to go beyond these
The paper articulates both the key
abstractions underlying NELL and its
limitations, which suggest avenues for
future work in its concluding discussion section.
In a world that has become obsessed with the latest deep neural network mechanism, and its performance
on one benchmark or another, NELL
is an important reminder of the power
another style of research: exploratory
research that seeks create new paradigms and substantially broaden the
capabilities and the sophistication of
machine learning systems.
1. Banko, M., Cafarella, M.J., Soderland, S., Broadhead,
M. and Etzioni, O. Open information extraction from the
Web. IJCAI, 2007.
2. Caruana, R. Multitask learning: A knowledge-based
of source of inductive bias. In Proceedings of the 10th
International Conference on Machine Learning. (San
Mateo, CA, USA, 1993). Morgan Kaufmann, 41–48.
3. Craven, M. DePasquo, D., Freitag, D., McCallum, A.,
Mitchell, T.M., Nigam, K. and Slattery, S. Learning to
extract symbolic knowledge from the World Wide
Web. In Proceedings of the AAAI/IAAI, 1998.
4. Etzioni, O. Deep learning isn’t a dangerous magic genie.
It’s just math. Wired (June 15, 2016); https://www.
5. Etzioni, O., Cafarella, M.J., Downey, D. Popescu, A-M,
Shaked, T. Soderland, S., Weld, D.S. and Yates, A.
Unsupervised named-entity extraction from the Web:
An experimental study (2005); http://bit.ly/2F5MZlf
6. Gibley, E. Google AI algorithm masters ancient game
of Go. Nature (Jan. 27, 2016); http://www.nature.com/
7. Thrun, S. and Mitchell, T.M. Learning one more thing.
Oren Etzioni is Chief Executive Officer of the Allen
Institute for Artificial Intelligence, Seattle, and professor
of computer science at the University of Washington,
Seattle, WA, USA.
Copyright held by author.
THE FIELD OF ARTIFICIAL INTELLIGENCE
(AI) is rife with misnomers and machine learning (ML) is a big one. ML
is a vibrant and successful subfield,
but the bulk of it is simply “function
approximation based on a sample.”
For example, the learning portion
of AlphaGo—which defeated the human world champion in the game of
GO—is in essence a method for approximating a non-linear function
from board position to move choice,
based on tens of millions of board
positions labeled by the appropriate
move in that position.a As pointed out
in my Wired article, 4 function approximation is only a small component of
a capability that would rival human
learning, and might be rightfully
called machine learning.
Tom Mitchell and his collaborators
have been investigating how to broaden the ML field for over 20 years under
headings such as multitask learning, 2
life-long learning, 7 and more. The following paper, “Never-ending Learning,” is the latest and one of the most
compelling incarnations of this research agenda. The paper describes
the NELL system, which aims to learn
to identify instances of concepts (for
example, city or sports team) in Web
text. It takes as input more than 500M
sentences drawn from Web pages, an
initial hierarchy of interrelated concepts, and small number of examples
of each concept. Based on this information, and the relationships between the concepts, it is able to learn
to identify millions of concept instances with high accuracy. Over time,
NELL has also begun to identify relationships between concept classes,
and extend its input concept set.
The NELL project is important and
unique for a number of additional
a Of course, this is an oversimplification but it
suffices for our purposes here. See AlphaGo in
Nature6 for an in-depth presentation.
1. The system has been running
at CMU for over five years, and its
knowledge base is available online
for inspection and download here:
2. The work is also an instance of
‘Reading the Web,’ a paradigm that
was inspired by Mitchell’s WebKB
project. 3 The paradigm led to the
KnowItAll system, 5 Open Information
Extraction, 1 and much more.
3. The paper both places the work in
context (“Learning in NELL as an approximation to EM”) and identifies key
lessons from the effort (“To achieve
successful semi-supervised learning,
couple the training of many different
As is often the case with outstanding research, the work raises many
open questions including:
1. Could one, with the benefit of
hindsight, reimplement NELL in a
radically more efficient fashion where
iterations of the learning process take
2. What is the end-state of NELL’s
3. While NELL taught us a lot about
continuously running semi-supervised
learning systems, it is still unable to perform increasingly challenging learning
tasks over time. What are the next steps
in the life-long learning paradigm?
the NELL system,
to learn to identify
instances of concepts
in Web text.
Breaking the Mold
of Machine Learning
By Oren Etzioni
To view the accompanying paper,