entering an address by sending it from the web to
the car [ 11].
All of these connected functions can potentially
be exploited by UE for research needs. For example,
variants of the software can be sent to target user
populations for either their subjective feedback or
to generate “A/B” test cases. A population’s configuration settings can be analyzed to determine the
most and least popular setting changes.
Most important, the software development organization can rapidly update the product during
usability testing and then observe users for their
reactions to the device and its new software.
Many Groups Use Back-haul Data
User experience is hardly the only group motivated
to scoop up back-haul data. Back-haul data collection is essential for other functions to fulfill their
The ability for UE to understand, leverage, and
champion these other needs will maximize its
influence over the form that back-haul infrastructure takes. In particular, UE needs to realize its
stake in the design of these infrastructure components when initial implementations are sketched. A
common mistake for typical small UE teams is not
seeing the potential at early stages.
For example, the operations group will require
usage data for authentication and billing. IT needs
aggregate usage metrics that trigger alarms when
usage levels drop, whether due to instability in its
own infrastructure or to service-provider outages.
Support needs some form of back-haul data to
understand customer issues more efficiently, such
as versioning, configuration settings, stack traces
[ 12], and prior usage. QA values back-haul data on
common performance metrics, such as “time until
first GPS fix” and “time to first network connection”
so that it can assess release readiness. QA also values back-haul data to help track metrics of system
performance and stability. With this data, the QA
team can better evaluate release readiness. Finally,
everyone wants some method to “file a problem
report” from the device to reduce the time to file,
reproduce, and fix bugs.
may say the product is performing as specified, but
they don’t indicate if the users are actually happy
with that level of performance.
For UE design, it is beneficial to triage use cases
into buckets such as “frequent for all,” “frequent for
some,” or “infrequent for all.” In order to do this, the
data source must maintain a marker of individuality along with the data, so the data analyst can slice
and dice the data to discover such relationships.
In addition, the UE group desires the ability to
run longitudinal queries, particularly the ability to
see that new user features lead to perceptible user
benefits. This requires the UE group itself to create
and maintain over time a database of high-level
user events in a normalized format.
[ 11] MyDash, Send2Car,
[ 12] Wikipedia,
Reporting ( WER),”
September + October 2009
Issues Unique to UE Needs
The UE group has needs distinct from the other
functional groups. Both QA and operations, by and
large, are more concerned with aggregate numbers
than an individual’s experience. Their numbers
In order to exploit the back-haul data, central sampling issues such as who, what, and when (how frequently) need addressing.
Behavioral logging benefits from the ability to
understand and filter the results through the lens
of various user-grouping mechanisms. For example,
stakeholders will question unwelcome results
because various outlier groups may have skewed
Invariably, both business and operational groups
have an interest in creating user segmentation; this
functionality is thus almost certainly available to
the UE group. However, the grouping mechanisms
will be driven by customer purchasing segments,
and this in turn can constrain an experiment’s
Problem reports. Since users have problems, they
need a mechanism to explain what those problems
are: the problem report. There are two different
Mostly passive, where the problem report is •
created on behalf of the users. They just need to
submit the report.
Mostly active, where the user initiates the prob- •
Both are, of course, useful. But for devices, the
active report is particularly useful, as it is often
very difficult to create bug reports independent
of a complex environment that only the user fully
The work required is not just the data transmission and recording. In implementing the problem
report feature, I recommend budgeting a fair
amount of time for implementing tools that facili-