ety, and thus should be trustworthy, it
is important to understand how trust
and social norms contribute to the success of human society.
“Trust is a psychological state comprising the intention to accept vulnerability based upon positive expectations
of the intentions or behavior
of another.” 28
Trust enables cooperation. Cooperation produces improved rewards. When
a group of people can trust each other
and cooperate, they can reap greater rewards—sometimes far greater rewards—than a similar group that does
not cooperate. This can be through division of labor, sharing of expenses,
economies of scale, reduction of risk
and overhead, accumulation of capital,
or many other mechanisms.
It is usual to treat morality and ethics as the foundations of good behavior, with trust reflecting the reliance
that one agent can have on the good
behavior of another. My argument
here inverts this usual dependency,
holding that cooperation is the means
by which a society gains resources
through the behavior of its individual
members. Trust is necessary for successful cooperation. And morality
and ethics (and other social norms)
are mechanisms by which a society
encourages trustworthy behavior by
its individual members.
As a simple example, suppose that
you (and everyone else) could drive anywhere on the roads. (This was actually
true before the early 20th century. 14) Everyone would need to drive slowly and
cautiously, and there would still be frequent traffic jams and accidents. With
a widely respected social norm for driving on the right (plus norms for intersections and other special situations),
transportation becomes safer and
more efficient for everyone. Obedience
to the social norm frees up resources
Like driving on the right, a huge saving in resources results when the people in a society trust that the vast majority of other people will not try to
kill them or steal from them. People
are able to spend far less on protecting themselves, on fighting off attacks, and on recovering from losses.
The society earns an enormous “peace
dividend” that can be put to other productive uses. 25 Through trust and co-the content of the relevant knowledge.
First, I define trust, and evaluate the
use of game theory to define actions.
Next, I explore an approach whereby an
intelligent robot can make moral and
ethical decisions, and identify open research problems on the way to this
goal. Later, I discuss the Deadly Dilemma, a question that is often asked
about ethical decision making by self-driving cars.
What is trust for? Society gains resources through cooperation among its
individual members. Cooperation requires trust. Trust implies vulnerability.
A society adopts social norms, which we
define to include morality, ethics, and
convention, sometimes encoded and
enforced as laws, sometimes as expectations with less formal enforcement, in
order to discourage individuals from exploiting vulnerability, violating trust,
and thus preventing cooperation.
If intelligent robots are to participate in our society—as self-driving
cars, as caregivers for elderly people or
children, and in many other ways that
are being envisioned—they must be
able to understand and follow social
norms, and to earn the trust of others
in the society. This imposes requirements on how robots are designed.
The performance requirements on
moral and ethical social norms are
quite demanding. ( 1) Moral and ethi-
cal judgments are often urgent, need-
ing a quick response, with little time
for deliberation. ( 2) The physical and
social environments within which
moral and ethical judgments are
made are unboundedly complex. The
boundaries between different judg-
ments may not be expressible by sim-
ple abstractions. ( 3) Learning to im-
prove the quality and coverage of
moral and ethical decisions is essen-
tial, from personal experience, from
observing others, and from being
told. Conceivably, it will be possible
to copy the results of such a learning
process into newly created robots.
Insights into the design of a moral
and ethical decision architecture for
intelligent robots can be found in the
three major philosophical theories of
ethics: deontology, utilitarianism,
and virtue ethics. However, none of
these theories is, by itself, able to meet
all of the demanding performance requirements listed here.
A hybrid architecture is needed, operating at multiple time-scales, drawing on aspects of all ethical theories:
fast but fallible pattern-directed responses; slower deliberative analysis of
the results of previous decisions; and,
yet slower individual and collective
Likewise, it will be necessary to express knowledge at different levels of
information richness: vivid and detailed perception of the current situation; less-vivid memories of previously
experienced concrete situations; stories—linguistic descriptions of situations, actions, results, and evaluations; and rules—highly abstracted
decision criteria applicable to perceived situations. Learning processes
can abstract the simpler representations from experience obtained in the
rich perceptual representation.
What Is Trust For?
If intelligent robots (and other AIs) will
have increasing roles in human soci-
Figure 1. The Prisoner’s Dilemma. 5
C – 1, – 1 – 4, 0
D 0, – 4 – 3, – 3
You and your partner are two prisoners who are separated and offered the following deal:
If you testify against your partner, you will go free, and your partner goes to jail for four
years. If neither of you testifies, you each go to jail for one year, but if you both testify,
you both get three years. The action C means “cooperate,” which in this case means
refusing to testify. The action C means “defect,” which refers to testifying against your
partner. The entries in this array are the utility values for (you, partner), and they reflect
individual rewards (years in jail).
No matter which choice your partner makes, you are better off choosing action D. The
same applies to your partner, so the Nash equilibrium (the “rational” choice of actions)
is (D, D), which is collectively the worst of the four options. To attain the much better
cooperative outcome (C, C) by choosing C, you must trust that your partner will also
choose C, accepting your vulnerability to your partner choosing D.