usability tests of an airline website. The
purpose was to compare approaches to
An overview of the key findings from
the CUE studies is shown in Table 2.
The following sections discuss these
findings in more detail.
Huge number of issues. The total
number of usability issues for the state-
of-the-art websites that we have tested
is huge. It is much larger than you can
hope to find in one usability test. For
• CUE- 2 found 310 usability
problems in Hotmail in 2001.
• CUE- 4 found 340 usability
problems on a hotel website in 2003.
• CUE- 9 found 223 usability
problems on a van-rental website in
The figures in Table 3 indicate
that all studies were far from finding
all usability issues. In particular, the
substantial number of issues that were
reported by single teams only indicate
that the 310, 340, and 223 usability
issues may just be the tips of the
If we had asked further teams to
participate in any of these studies,
or if the participating teams had run
additional test sessions with different
test tasks, additional problems would
have been found.
Takeaway: Never claim or assume
that you can carry out an exhaustive
usability test of a website. Exhaustive
testing may be possible within limited
Five users are not enough. It is
a widespread myth that five users
are enough to find 85 percent of the
usability problems in a product. As
shown in Table 3, the CUE studies have
consistently shown that even 15 or
more professional teams report only a
fraction of the problems.
If you vary the task set, the
moderator, or the usability test
procedure, new problems will be
found. Many serious or critical
problems cannot be discovered
by a particular moderator or by a
particular task set [ 6].
Takeaway: Five users—or 20
users, or even 100 users—will find
only a small fraction of the usability
problems. The last row in Table 3 shows
that they will not even find all serious
or critical problems. Nothing in the
CUE studies contradicts, however,
that five users may be enough to drive a
useful iterative process.
No gold standard. CUE results
consistently show that usability testing
is not the expensive, high-quality
gold standard against which all other
methods can be measured.
Takeaway: Use usability testing as
Never claim or
assume that you
can carry out an
test of a website.