relayed on the right. Timings and text
are exactly as they were in the test.
So, could you “pass the test” and be
able to say which of the two entities—
E20 and E24—is the human and which
Huma Shah, London, U.K., and
Kevin Warwick, Reading, U.K.
The details of this 2014 Turing Test
experiment only reinforces my judgment
that the Turing Test says little about
machine intelligence. The ability to
generate a human-like dialogue is at best an
extremely narrow slice of intelligence.
Moshe Y. Vardi, Editor-in-Chief
WE WISH TO clarify an ac- count of the 2014 Tur- ing Test experiment we conducted at the Royal Society London, U.K.,
as outlined by Moshe Y. Vardi in his
Editor’s Letter “Would Turing Have
Passed the Turing Test?” (Sept. 2014).
Vardi was referring to a New Yorker blog
by Gary Marcus, rather than to our experiment directly. But Marcus had no firsthand experience with our 2014 experiment nor has he seen any of our Turing
Our experiment involved 30 human
judges, 30 hidden humans, and five
machines—Cleverbot, Elbot, Eugene
Goostman, JFred, and Ultra Hal; for background and details see http://turingtest-
We used social media to recruit judges
and a variety of hidden humans, including males, females, adults, teenagers, experts in computer science and robotics,
and non-experts, including journalists,
lecturers, students, and interested members of the public.
Prior to the tests, the judges were unaware of the nature of the pairs of hidden
entities they would be interrogating; we
told them only that they would simultaneously interrogate one human and one machine for five minutes and that the human
could be a male or female, child or adult,
native English speaker, or non-native English speaker. We asked the hidden humans
to be themselves, that is, to be human.
The 30 judges, each given an anonymous experiment identity—labeled J1–
J30—interrogated five pairs of hidden entities. Likewise each human and machine
was given a unique identity—E1–E35. We
ran 150 “simultaneous comparison” Turing Tests in which we instructed the judges that their task was to determine which
was human and which was machine in
the pair, a decision to be made based
solely on the responses the hidden entities posted in reply to what a judge said.
Eugene Goostman was not correctly
identified as the machine in the pair
in 10 of its 30 tests; that is, 10 judges
did not recognize it was a machine. Eu-
gene Goostman’s personality is that of
a 13-year-old boy from Odessa, Ukraine,
a character we do not consider contrary
to Alan M. Turing’s vision for build-
ing a machine to think. In 1950, Tur-
ing said, “Instead of trying to produce a
programme to simulate the adult mind,
why not rather try to produce one which
simulates the child’s?”
The figure here includes one simul-
taneous conversation from the experi-
ment, showing one of Judge J19’s tests
after that judge simultaneously inter-
acted with two hidden entities, in this
case E20 and E24. In this test, E20’s
responses to the judge were relayed to
a message box displayed on the left of
the judge’s screen; E24’s answers were
Human or Machine?
Simultaneous comparison by judge J19 in session four, round one of hidden entities E20
and E24 in a Turing Test.
[11:59: 14] J19: Hello
[11:59: 18] E20: I was hoping to see you
[11:59: 25] J19: good!
[11:59: 29] E20: That's good to hear.
[11:59: 38] J19: how are you
[11:59: 41] E20: Everything's fine, Caused
any scandals lately?
[12:00:09] J19: no not really!
[12:00: 22] E20: Can you use better
grammar and rephrase your question? I'd
like to help, but I can't even make it through
the jumbled mess. Use capitals and
[12:00: 34] J19: sorry
[12:00: 38] E20: You're excused.
[12:00: 49] J19: 't type as good as I talk
[12:00: 55] E20: Talk to me whenever you
[12:01: 49] J19: Are you interested in
[12:01: 55] E20: I believe machines should
do many to assist the lives of man.
[12:02: 12] J19: I agree
[12:02: 17] E20: You are being well behaved
[12:02: 49] J19: as well behaved as I can be!
[12:03:02] E20: Well then our peanut
distracted me from the movie. Bad baby. [I
look at your stomach as I say it, laughing] I
don't mind if -.
[12:03: 19] J19: you're funny!
[12:03: 23] E20: Do you really think so? I'll
always believe you're brilliant!
[12:04:00] J19: thankyou remote19 left
[12:04:03] E20: Don't think anything of it!
[11:59: 57] J19: How are you doing?
[12:00:07] E24: I'm doing fine, you?
[12:00: 17] J19: great
[12:00: 27] E24: That's good. :)
[12:01:07] J19: are you enjoying this process?
[12:01: 17] E24: So far seems good.
[12:02:02] J19: sorry if I am slow
[12:02: 12] E24: That's ok.
[12:02: 31] J19: do you have a favorite tv
[12:02: 47] E24: I have quite a few.
[12:03:01] J19: me too
[12:03: 16] E24: What type ot TV do you like?
[12:03: 34] J19: Game of thrones
[12:03: 50] E24: I prefer documentories.