I M A G E B Y B L O S S O M S T A R / S H U T T E R T O C K . C O M

usability tests of an airline website. The purpose was to compare approaches to test moderation.

 

KEY FINDINGS

An overview of the key findings from the CUE studies is shown in Table 2. The following sections discuss these findings in more detail.

Huge number of issues. The total
number of usability issues for the state-
of-the-art websites that we have tested
is huge. It is much larger than you can
hope to find in one usability test. For
example:
• CUE- 2 found 310 usability
problems in Hotmail in 2001.

• CUE- 4 found 340 usability problems on a hotel website in 2003.

• CUE- 9 found 223 usability problems on a van-rental website in

2011.

The figures in Table 3 indicate
that all studies were far from finding
all usability issues. In particular, the
substantial number of issues that were
reported by single teams only indicate
that the 310, 340, and 223 usability
issues may just be the tips of the
icebergs.

If we had asked further teams to participate in any of these studies, or if the participating teams had run additional test sessions with different test tasks, additional problems would have been found.

Takeaway: Never claim or assume that you can carry out an exhaustive usability test of a website. Exhaustive testing may be possible within limited function areas.

Five users are not enough. It is
a widespread myth that five users
are enough to find 85 percent of the
usability problems in a product. As
shown in Table 3, the CUE studies have
consistently shown that even 15 or
more professional teams report only a
fraction of the problems.

If you vary the task set, the moderator, or the usability test procedure, new problems will be found. Many serious or critical problems cannot be discovered by a particular moderator or by a particular task set [ 6].

Takeaway: Five users—or 20 users, or even 100 users—will find only a small fraction of the usability problems. The last row in Table 3 shows that they will not even find all serious or critical problems. Nothing in the CUE studies contradicts, however, that five users may be enough to drive a useful iterative process.

No gold standard. CUE results consistently show that usability testing is not the expensive, high-quality gold standard against which all other methods can be measured.

Takeaway: Use usability testing as

Never claim or
assume that you
can carry out an
exhaustive usability
test of a website.