It Can Be Achieved—A Real
Case Example
One successful example from my
tenure at Oracle is worth sharing. It demonstrates both the way
data can be harnessed to motivate
investment, and, incidentally, a
clever interpretation of UX metrics
by Oracle’s famous CEO and founder, Larry Ellison.
In the late 1990s, Oracle had
an active and highly quantitative
usability measurement program in
place that was executing between
80 and 100 user research evaluations per year. After some initial
harsh customer and press feedback on a new CRM application, I
was asked to present any usability
information we had prior to product launch. This, of course, put
the GM of this division under the
microscope, because the data was
not flattering. However, the boardroom discussion quickly turned to
how this product compared with
other products in the company
portfolio. Fortunately, we had two
years of consistent data spanning
most product lines and ranking
them on average task completion,
error rates, and satisfaction. The
data was in what today would be
recognized as a precursor to the
ANSI/NIST Common Industry
Format because the architect of
Oracle’s usability measurement
program, Anna Wichansky, was
one of the original leaders of the
NIST CIF effort. This international
standardization effort for usability
data was completed several years
after the incident I am describing.
In addition to success and time-on-task data, several approaches to
measuring satisfaction were used,
which allowed for comparison
and trend tracking. One of them
was SUMMI—a favorite in the
boardroom because of its ability to
reference an anonymous industry
What Is at Stake for You?
While this particular incident
offers a glimpse of hope, what
happens when the UX leader does
not have an ongoing usability and
evaluation program that meets the
CEO litmus test? The answers to
questions 2 and 3 raised earlier are
not pretty. The result of neglecting
the information needs of executives is often that, from their perspective, usability becomes viewed
as a soft, non-scientific data
source loosely equivalent to the
anecdotal viewpoints traded every
day as fact by executives, who
are by nature comfortable making important business decisions
with some degree of ambiguity.
In the end, this trickles down to
limit the business effectiveness of
both the UX team and its leadership. As we look in the historical
mirror of our evaluation practice,
we need to remain cognizant of
this CEO gap and make an effort
to close it for the long-term benefit of both our profession and
our companies’ shareholders.
The potential upside if the gap
can be closed is huge. Conversely,
the potential downside is never
reaching the professional status of more mature disciplines
in the business world, such as
marketing and engineering,
who bring their own supporting
evaluation data and metrics into
the boardroom to do battle for
their point of view every day.
average. SUMMI is an extensive
questionnaire-based approach used
in the 1990s that yielded a score
between one and 50, where 50 was
the best quality value.
For the following two years, we
reported in person in the boardroom each week’s usability lab
data. Larry often demanded retests
of products before each release as
a method to drive usability decisions deep into the planning process. He also turned this data into
an internal competitive motivator
for direct reports and frequently
threatened to base their bonuses
on it. (I don’t know if he actually
ever followed through.)
What was most interesting was
how SUMMI was internalized intellectually in the organization. In
the second year of these weekly
boardroom usability evaluation
reviews, Charles Phillips joined
the company as co-president and
attended his first review session.
He was not familiar with usability metrics at all, and I was not
expecting him to be in the meeting. Generally, it is best to start
someone’s introduction to usability
with a lab tour, not in a high-stakes
board meeting. As I mentally prepared to give an impromptu (short)
metrics definition lecture with
the CEO and a room full of development EVPs, Larry interrupted
me and explained all the metrics, one by one, himself. Having
a background in mathematics,
he not only understood but also
internalized the relevance and
models behind them to the business. When he began to explain
SUMMI, he looked at Charles and
said, “This one is easy: Just double
the number and think of it as the
IQ of the product. It has to be near
100, because we are not going to
ship any dumb products. Doing
so is detrimental to our brand.”
ABOUT THE AUTHOR
Dan Rosenberg recently founded
rCDO UX LLC, a UX strategy firm
serving C-level executives and
industry UX leaders in defining
competitive design strategies and
executing them. He led global UX
March + April 2013
DOI: 10.1145/2427076.2427092
© 2013 ACM 1072-5520/13/03 $15.00
