patch to a trained ML component that
classifies the patches into different
categories, which are then shown in a
sorted gallery. Each defined class in
the trained model is shown as a patch
in the same gallery, and the user can
then 1) click on a patch to see it in the
main view to get a sense of its context
in the tissue, and 2) change a label by
either dragging the patch to the correct
category or by clicking on the button or
the corresponding shortcut key. Note
that, similar to the previous example,
the output of the classifier is limited to
producing predictions not for the entire
area, but only for a representative
systematic sampling of that area, which
makes the validation and correction
effort tractable.
Like the mitotic counter in the
previous section, both systems share
the property that the generated
parameter that the clinician wants
to assess can be derived from manual
input only. If the nuclei-detection
algorithm failed to detect any nuclei,
the user could still manually click on
all the nuclei to calculate the KI- 67
index. However, the amount of clicking
would likely overwhelm the user. These
user-correction systems do not strictly
need an ML component, but practical
usability requires automated support
with a certain level of prediction
accuracy.
Another crucial factor when
designing this type of system is that the
user-correction accuracy needs to be
higher than that of the ML component
alone, in order for the generated
training data to add value when
retraining the ML component.
DISCUSSION
In the design of these tools, we have
paid special attention to ensuring
that manual, unassisted workflows
are preserved and compatible with
the assisting tools. When possible,
training-data collection has been
designed to become an integrated
part of clinicians’ daily diagnostic
practice. Furthermore, as the
predictive performance of our initial
ML components improve using the
clinicians’ corrections as training data,
we look forward to the point when our
user interfaces need to be redesigned or
augmented with interactions that are
adapted to ML components with much
higher performance. It is our ambition
to design the new interfaces so that
the user can provide corrections and
simultaneously teach and verify results
at increasingly higher levels, forming
a verification staircase [ 4], as opposed
to a steep cliff where the user has to
validate all or nothing.
Looking forward, as the technology
comes of age, we hope this human-centered teaching aspect will continue
to spark novel designs as interactions
expand from dyadic teacher-learners
to networks of machine intelligences
DOI: 10.1145/3282860 COP YRIGHT HELD BY AUTHORS
and groups of human specialists co-evolving, teaching, and learning in
daily practice.
Observing ML-based product development from the view of training-data
generation, we have shown how decisions made by the UX designer have an
enormous impact on project success.
Each step of training-data generation
needs to get the motivations right so that
users are willing and able to provide corrections. The choice of what the training
dataset should consist of and thus what
the ML component should predict
is tightly connected to how the user
interface should look, behave, and be
interacted with. Hence, in the creation
of human-centered machine-learning
systems, the UX designer plays a key role
from the start and throughout.
Endnotes
1. McGuinness, K. and O’Connor, N.E. A
comparative evaluation of interactive
segmentation algorithms. Pattern
Recognition 43, 2 (2010), 434–444.
2. von Ahn, L. and Dabbish, L.. Labeling
images with a computer game. Proc. of
the SIGCHI Conference on Human Factors
in Computing Systems. ACM. New York,
2004, 319–326.
3. Shneiderman, B. The future of interactive
systems and the emergence of direct
manipulation. Behaviour & Information
Technology 1, 3 (1982), 237–256.
4. Molin, J., Woźniak, P. W., Lundström, C.,
Treanor, D., and Fjeld, M. Understanding
design for automated image analysis in
digital pathology. Proc. of the 9th Nordic
Conference on Human-Computer Interaction.
ACM, New York, 2016, Article 58.
Martin Lindvall is an industrial Ph.D.
student exploring interactions using machine
learning as a design material. Currently part of
the Wallenberg AI, Autonomous Systems and
Software Program ( WASP), his background
includes an M.Sc. in cognitive science and 10
years of experience designing and developing
medical information systems at Sectra.
→ martin.lindvall@sectra.com
Jesper Molin is research scientist and UX
designer at Sectra, exploring and designing
ML-based tools used within clinical routine
pathology. His background includes an M. Sc. in
applied physics and electrical engineering and
a Ph.D. in human-computer interaction from
Chalmers University of Technology.
→ jesper.molin@sectra.com
Jonas Löwgren is professor of interaction
and information design at Linköping University,
Sweden. His expertise includes collaborative
media, interactive visualization, and the design
theory of digital materials.
→ jonas.lowgren@liu.se
Figure 6. Patch gallery prototype, samples from the tissue are generated and classified by an
ML algorithm into three classes, which the user can correct by drag ’n drop.