Interactions - November/December 2018

means that assessing and addressing bias needs to be part of the product team’s normal goal-setting processes, while in-depth research potentially has to be resourced separately so it does not weigh on teams’ local roadmaps. Education efforts have to be structured in a manner that doesn’t appear as yet another training. In our pilot, we started to include algorithmic bias in existing machine-learning courses for different internal audiences, and in both engineering and diversity events, rather than requiring special attendance.

Companies can highlight the importance of algorithmic-bias efforts by including it in company-wide baseline expectations combined with specific individual team goals, by resourcing the simplification of tools to fit company-specific practices, and by making (future) dashboards and meta-datasets widely available. This does benefit from leadership buy-in, which requires quick executive summaries on why algorithmic bias is important and how to best address it organizationally. Different arguments will convince different audiences, and a step-by-step approach is crucial, rather than a purist approach that strives to solve everything all at once or that potentially punishes developers for sharing issues that they encounter.

INCREASING IMPACT AS A RESEARCH COMMUNITY

Great work is being done in the HCI
and machine-learning communities to
address algorithmic bias; however, in
practice it’s challenging to consume
all of it and even harder to translate
the discussions and findings into
practical approaches for product
teams. Continually following the
scientific literature on fairness
requires specific expertise and focus
not always available in production
settings. Turning research literature
into guidelines applicable in practice
requires a translation that cannot
be expected from all paper authors.
Applied industry researchers
absolutely must be able to fulfill this
mediating role, hopefully aided by
more accessible literature.
Easy-to-use summaries of papers
and the informal sharing of concrete
experiences with both successful and
problematic results is useful in better
understanding what gaps to close, and
how to best have impact. For example,
when inevitable discussions on “what
fair means” arise, summaries such
as the 21 Definitions of Fairness [ 11]
are invaluable points of reference.
When setting up an effort to assess
and address bias, overviews of how
other companies are approaching
the issue are especially helpful and
can motivate prioritization and
resourcing. This also means that
we need to be open to sharing less-
than-ideal learning experiences. In
the majority of cases, teams want to
make the right decisions but don’t
necessarily know how—let’s help
make this easier. In this article,
we have offered insights into our
approach to doing so.

Endnotes

1. Friedman, B. and Nissenbaum, H. Bias in computer systems. ACM Trans. Inf. Syst.

14, 3 (1996), 330–347.

2. FATML; https://www.fatml.org/ resources/principles-for-accountable-algorithms

3. ACM Conference on Fairness,
Accountability, and Transparency
(ACM FAT*); https://fatconference.org

4. AI Now institute. Algorithmic impact assessments: A practical framework for public agency accountability. Apr. 2018; https://ainowinstitute.org/ aiareport2018.pdf

5. Data & Society. Algorithmic accountability primer. Apr. 2018; https://datasociety.net/wp-content/ uploads/2018/04/Data_Society_ Algorithmic_Accountability_Primer_ FINAL- 4.pdf

6. World Wide Web Foundation. Algorithmic accountability report, 2017; https://webfoundation.org/docs/2017/07/ Algorithms_Report_WF.pdf

7. Gebru, T., Morgenstern, J., Vecchione, B., Wortman Vaughan, J., Wallach, H., Daumeé III, H., and Crawford, K. Datasheets for datasets. 2018; https:// arxiv.org/abs/1803.09010

8. Olteanu, A., Castillo, C., Diaz, F., and Kiciman, E. Social data: Biases, methodological pitfalls, and ethical boundaries. 2016; http://dx.doi. org/10.2139/ssrn.2886526

9. Baeza-Yates, R. Data and algorithmic bias in the web. Proc. of the 8th ACM Conference on Web Science.

ACM, New York, 2016; https://doi. org/10.1145/2908131.2908135

10. Springer, A. and Cramer, H. “Play PRBLMS”: Identifying and correcting less accessible content in voice interfaces. Proc. of CHI ’ 18. ACM, New York, 2018.

11. Narayanan, A. FAT* 2018 tutorial: 21 fairness definitions and their politics.

Henriette Cramer is a lab lead at Spotify, where her research has focused on voice interactions and road managing Spotify’s algorithmic accountability effort. She is especially interested in data and design decisions that affect machine-learning outcomes, and the (mis)match between machine models and people’s perceptions. → henriette@spotify.com

Jean Garcia-Gathright is a machine-learning engineer and researcher at Spotify, where she specializes in the evaluation of data-driven models that power engaging, personalized user experiences. → jean@spotify.com

Aaron Springer is a Ph. D. candidate at University of California Santa Cruz. His research focuses on the user experience of machine learning, including fairness, trust, and transparency. → alspring@ucsc.edu

Sravana Reddy is a researcher at Spotify working in natural language processing and machine learning. Her interests lie in computational sociolinguistics, NLP for creative language, audio processing, and of late, fairness and bias in ML. She received her Ph.D. from the University of Chicago. → sravana@spotify.com DOI: 10.1145/3278156 COP YRIGH T HELD BY AUTHORS. PUBLICATION RIGHTS LICENSED TO ACM. $15.00