operation, the society becomes healthier and wealthier.
Castelfranchi and Falcone11 define
trust in terms of delegation, and the
agent’s confidence in the successful
performance of the delegated task.
They provide clear and valuable definitions for the trust relationship between
individuals. However, there is also a
role for invariants that individuals can
trust holding across the society (for example, no killing, stealing, or driving
on the wrong side of the road), and the
role of individual behavior in preserving these invariants.
Game theory: Promise and problems.
We might hope that progress in artificial intelligence (AI) will provide technical methods for achieving cooperation
and trustworthiness in a robot. The leading textbook in AI appeals to decision
theory to tell us that “a rational agent
should choose the action that maximizes
the agent’s expected utility” 29
The utility term U(s) represents the individual agent’s preference over states of
the world, and e is the available evidence.
The agent’s knowledge of the “physics of
the world” is summarized by the probability term P (RESULT (a) = s′ |a, e).
Game theory is the extension of decision theory to contexts where other
agents are making their own choices to
maximize their own utilities. 20 Game
theory asserts that a vector of choices
by all the agents (a strategy profile) can
only be a “rational choice” if it is a Nash
equilibrium—that is, no single agent
can improve its own utility by changing
its own choice (often reducing utilities
for the others).
Utility U(s) is the key concept here.
In principle, utility can be used to represent highly sophisticated preferences,
for example, against inequality or for
increasing the total welfare of everyone
in the world. 32 However, sophisticated
utility measures are difficult to implement. Typically, in practice, each
agent’s utility U(s) represents that individual agent’s own expected reward.
In recreational games, this is rea-
sonable. However, when game theory
is applied to model the choices indi-
viduals make as members of society, a
simple, selfish model of utility can
yield bad results, both for the individu-
al and for the society as a whole. The
Prisoner’s Dilemma5 is a simple game
(see Figure 1), but its single Nash equi-
librium represents almost the worst
possible outcome for each individual,
and absolutely the worst outcome for
the society as a whole. The cooperative
strategy, which is much better for both
individuals and society as a whole, is
not a Nash equilibrium, because either
player can disrupt it unilaterally.
The Public Goods Game26 is an N-
person version of the Prisoner’s Di-
lemma where a pooled investment is
multiplied and then split evenly
among the participants. Everyone
benefits when everyone invests, but a
free rider can benefit even more at ev-
eryone else’s expense, by withholding
his investment but taking his share of
the proceeds. The Nash equilibrium
in the Public Goods Game is simple
and dystopian: Nobody invests and no-
These games are simple and ab-
stract, but they capture the vulnerability
of trust and cooperation to self-interest-
ed choices by the partner. The Tragedy
of the Commons15 generalizes this re-
sult to larger-scale social problems like
depletion of shared renewable resourc-
es such as fishing and grazing opportu-
nities or clean air and water.
Managing trust and vulnerability.
Given a self-interested utility function,
utility maximization leads to action
choices that exploit vulnerability, elim-
inate trust among the players, and
eliminate cooperative solutions. Even
the selfish benefits that motivated de-
fection are lost, when multiple players
defect simultaneously, each driven to
maximize their own utility.
When human subjects play simple
economic games, they often seem to op-
timize their “enlightened self-interest”
rather than expected reward, trusting
that other players will refrain from ex-
ploiting their vulnerability, and often
being correct in this belief. 17, 39 Many
approaches have been explored for de-
fining more sophisticated utility mea-
sures, whose maximization would cor-
respond with enlightened self-interest,
including trust responsiveness, 6 credit
networks, 12 and augmented stage
games for analyzing infinitely repeated
Trust is necessary
ethics (and other
social norms) are
which a society
behavior by its