There are a range of devices that bridge the gap between user and application—from mice and keyboards, to screen readers, to specialized equipment that adapts the user interface for people with severe impairments. Unless we have had experience with these tools, it is difficult to conceive of how they work in practice and how they affect the user’s experience with our application. Furthermore, developers of UI tools seldom provide functional interfaces to support test automation or screen readers— making both challenging to implement.

In my view, usability testing is not inherently difficult, but it tends to be time consuming and hard to scale when it requires human observation of the people using the software being assessed. Also, because software developers seem to find the work fiddly or extraneous, they may fail at the first hurdle: deciding whether the testing effort is worth the investment.

Difficulties in test Automation
In addition to the basic challenges
in usability and accessibility testing,
there are also challenges in develop-
ing good automated testing frame-
works. Although pockets of interest-
ing academic research exist that focus
on automated testing, as an industry
we’ve found it difficult to translate
academic work into practical value.
Sharing of knowledge and tools from
academia has been limited, as com-
panies need first to pay for access to
many of the papers and then translate
the formal structure of those papers
into something that makes sense to
them. These solutions must also ad-
dress the expectations implicit in the
question, “Will this solve some of my
immediate issues today?” Complicat-
ing things is that commercial auto-
mated-testing tool providers tend to
guard their tests and methods, as they
perceive this to be to their competi-
tive advantage.

figure 1. screenshot of Google search keyboard navigation test.

examples of Automated testing

In 2009, I helped test several global software applications at Google. They were constructed using GWT (Google Web Toolkit), a very powerful development framework that allows developers to create cross-browser Web applications and hides many of the complexities from the developers. The resulting applications looked sleek and modern, and each had a user base of millions of people.

Development teams estimated GWT saved person-years of work in getting their applications into production. The way in which the teams were using GWT, however, resulted in several side effects that ended up generating usability and accessibility problems, such as broken keyboard navigation and poor support for screen readers. We discovered similar issues for other applications that used other frameworks and approaches, indicating these problems