embedded in the algorithms that make
or structure real-world decisions.
We model algorithm development,
implementation, and use as having five
distinct nodes—input, algorithmic operations, output, users, and feedback.
Importantly, we incorporate users because their actions affect outcomes. As
shown in the accompanying figure, we
identify nine potential biases. They are
not mutually exclusive, as it is possible
for multiple, interacting biases to exist
in a single algorithmic process.
Types of Bias
Training Data Bias. Predictive algorithms are trained on datasets, thus
any biases in the training data will be
reflected in the algorithm. In principle,
this bias should be easy to detect, but
the sources may be difficult to detect.
Presumed gold standard datasets, such
as government statistics or even judicial conviction rates, frequently contain bias. For example, if the criminal
justice system is biased, then, absent
corrections, the algorithm will mirror
such bias. Thus, training sets can be
subtle contributors to bias.
Algorithmic Focus Bias. Algorith-
mic focus bias occurs from both the
inclusion and exclusion of particular
variables. For instance, the exclusion
of gender or race in a health diagnos-
tic algorithm can lead to inaccurate or
even harmful conclusions. However,
the inclusion of gender, race, or even
ZIP codes in a sentencing algorithm
“code” functions like law in structur-
ing human activity. Algorithms and on-
line platforms are not neutral; they are
built to frame and drive actions.
Algorithmic “machines” are built
with specific hypotheses about the
relationship between persons and
things. As techniques such as machine
learning are more generally deployed,
concerns are becoming more acute.
For engineers and policymakers alike,
understanding how and where bias
can occur in algorithmic processes can
help address it. Our contribution is the
introduction of a visual model (see the
accompanying figure) that extends pre-
vious research to locate where bias may
occur in an algorithmic process.
Interrogating Bias in
Of course, social bias has been long recognized. Some attribute the introduction of bias into algorithms to the fact
that software developers are not well
versed in issues such as civil rights and
3 Others suggest it is far more
deeply embedded in society and its
4 Inspired by value chain
research, while our model cannot resolve bias; it provides a template for
identifying and addressing the sources
of bias—conscious or unconscious—
that might infect algorithms. What is
certain is that without proper mitigation, preexisting societal bias will be
and Ethnic Bias
How computing platforms and algorithms can potentially
either reinforce or identify and address ethnic biases.
societal bias will
be embedded in the
make or structure