egories, or brands, and tries to predict
the likelihood of purchase, engagement,
˲ User research aims at improving
the user experience by understanding the relation between actual usage
behaviors and the properties of the
design. To this end, it measures the
behavior and attitudes of users thereby
learning whether the product (or service) is usable, useful and delightful,
including after decision to purchase.
We urge organizations to act strategically and connect market research,
user research, and customer success
functions. This requires aligning goals
and sharing data among Marketing,
Sales, Customer Success, and the UX
Team (typically in Product or R&D). 1, 4
The Way Forward:
Educate Managers and Agile
We have shown five different ways that
agile teams without user research are
prone to building the wrong product.
To avoid such failures, we invite software managers and product teams
to assess and fill the current gap in a
team’s competencies. The closing table gives short-term and longer-term
action items to address the gaps.
1. Buley, L. The modern UX organization. Forrester
Report. (2016); https://vimeo.com/121037431
2. Grudin J. From Tool to Partner: The Evolution of Human-Computer Interaction. Morgan & Claypool, 2017.
3. HP report. Agile Is the New Normal: Adopting Agile
Project Management. 4AA5-7619ENW, May 2015.
4. Kell, E. Interview by Steve Portigal. Portigal blog.
Podcast and transcript. (Mar. 1, 2016); http://www.
5. Klein, L. UX for Lean Startups: Faster, Smarter User
Experience Research and Design. O’Reilly, 2013.
6. Loranger, H. UX Without User Research Is Not UX.
(Aug. 10, 2014) Nielsen Norman Group blog. http://
7. Mironov, R. Four Laws Of Software Economics. Part 2:
Law of Build Once, Sell Many. (Sept. 14, 2015); http://
8. Pear, R. Contractors Describe Limited Testing of
Insurance Web Site. New York Times (Oct. 24, 2013);
9. Perez, S. Users have low tolerance for buggy apps.
Techcrunch. (Mar 12, 2013);[ http://tcrn.ch/Y80ctA
10. Rosenberg, D. Introducing the business of UX.
Interactions. Forums. XXI. 1 Jan.–Feb. 2014.
11. Spool, J.M. Assessing your team’s UX skills. UIE. (Dec.
10, 2007); https://www.uie.com/articles/assessing_
Gregorio Convertino ( firstname.lastname@example.org)
is a UX manager and principal user researcher at
Nancy Frishberg ( email@example.com) is a UX researcher
and strategist, in private practice, and a 25+-year member
of the local SIGCHI Chapter BayCHI.org.
Copyright held by authors.
Confusing business leaders with users or the sales demo with the product
prototype leads companies to build
products based on what sales and
product managers believe is awesome
(for example, see Loranger6). Instead,
we advocate validating the designs
with actual end users during the product development.
Big data (What? When?) < The
full picture (... How? Why?).
Collecting and analyzing big data about
digital product use is popular among
product managers and even software developers, who can now learn
what features get traction with users. We support the use of big data
techniques as part of user research
and user-centered design, but not as
a substitute for qualitative user research. Let’s review two familiar ways
to use big data on usage: user data
analytics and A/B testing.
User data analytics can quickly answer questions about current usage:
quantity and most frequent patterns,
such as How many? How often?
When? Where? Once a product team
has worked out most of the design
(interaction patterns, page layouts,
and more), A/B testing compares design alternatives, such as “which image on a page produces more click-throughs”? In vivo experiments with
sufficient traffic can generate large
amount of useful data. Thus, A/B
testing is very helpful for small incremental adjustments.
Every software company is in the
business of finding and keeping new
customers. Suppose the logs show the
subscribers of an online dating application are not renewing. Should the company rejoice or despair? If people are
getting good matches, and thus are satisfied, non-renewal implies success. If
they are hopelessly disappointed by not
getting dates, non-renewal implies failure. Big data won’t tell you which, but
observing and listening to even a handful of non-renewing individuals will.
In brief, quantitative data is useful but has two limitations: First, it
will not tell the team why the current
features are or are not used. 5 Different
classes of users can have different reasons. Second, it will not identify what
additional or alternative features appeal to a new class of users unfamiliar with the product. To answer these
questions the team needs to rely on
qualitative research with existing and
proposed classes of users.
Market Research ≠ User Research
Finally, we point to the growing and
worrisome tendency in industry to mix
up user research with market research.
Market research groups make great
partners for user research. While user
research and market research have a few
techniques in common (for example,
surveys and focus groups), the goals and
variables they focus on are different.
˲ Market research seeks to understand attitudes toward products, cat-
Actions to address gaps in UX competencies.
1. Analyze the current skills of the team and
flag the gap. A functional product team needs
several key skill sets or UX competencies:
UX research, UX design, UI software
development and prototyping. 11 These might be
filled by training the current team members or
hiring UX professionals full-time or part-time.
2. Support product managers (or product
owners) with investment in UX.
Too often, product managers find their role
is a sort of “kitchen sink” for any task
that is not software development.
We encourage product managers to find
additional resources in the UX competencies,
to benefit both product and their workload.
3. Integrate UX competencies
a. Teams need UX research competencies as well as UX design skills (interaction, visual).
Other related skill sets include content development and documentation; accessibility;
globalization and localization.
4. Collect and prioritize findings from user research
a. Seek user feedback early and often.
b. Create channels to learn from end users and appropriate surrogates.
c. Prioritize UX issues during backlog grooming; remove friction and measure delight.
d. Build new features only after steps 4.a.–c. are done for each key version of the product.