The new advice given here is in three parts: We consider the stage before the evaluation, some tips for during the evaluation, and include guidelines for wrapping up the evaluation.
Planning the evaluation. Because children are, at the same time, predictable and unpredictable, it is important to plan well. In particular, carrying out a pilot evaluation that mimics as closely as possible the real evaluation is valuable. This pilot will demonstrate if the chosen recording methods are sensible, if any test tasks are doable, and if any survey instruments are age-appropriate.Before the real evaluation of the technology, there are often some design-and-create activities to be completed. Logging sheets for evaluators, survey instruments for use with the children, or diaries for the evaluation process might all need designing and piloting. Once these tasks are complete, there will also be a need to fix up transport, obtain consent from the children and their guardians, book rooms, arrange refreshments, and carry out a risk assessment.
Different locations and different methods. As interactive technology has become more mobile, and as schools have become more open to interactive technology, evaluations in labs are now quite rare. When looking at technology in schools, it is necessary to work within the structure and confines of the school day. The lesson length, for instanc e, is often an impermeable feature around which the evaluator will need to plan. Outdoor evaluations are difficult to control; our advice is to keep the evaluation as simple as possible, rely as little as you can on the use of technology, and carry out a very careful risk assessment. In many locations there will need to be a bad-weather backup plan. Four methods that have been studied in some depth over the past 10 years are diary methods, think-aloud methods, surveys, and the Wizard of Oz method. Diary methods are well suited to home evaluations and those that take place over a length of time; think aloud—previously assumed to be unusable with younger children—has been shown to be possible with children as young as seven and eight; and for surveys, many evaluators now use the fun toolkit, which is a validated method for gathering children’s opinions of technology. The location and maturity of an evaluation can dictate the method used. The use of diaries, for example, can be a good choice for evaluations at home, and Wizard of Oz studies (where children interact with a partially
functional product that is in part “driven” by an unseen assistant) can be very handy when a fully working system is not available. It should be noted that Wizard of Oz studies present some ethical issues, especially in relation to the use of deception. After the event, researchers will need to tell children that deception has occurred and give them the opportunity to withdraw their consent; where at all possible, open configurations should be used with the wizard seen.
Wrapping up. After evaluations, researchers need to thank the children and tell them what they’ve contributed. As outlined earlier, the child who participates in an evaluation has some right to know what the point was. While thanking the child, the researcher must often thank teachers and parents, and the more information that they can share about the nature and purpose of the evaluation, the better.
Back at the lab, the modern-day evaluator can breathe a huge sigh of relief (once any data has been made anonymous and safely stored and tagged) after what is often a noisy, but very enjoyable, day’s work. The information gathered about the interactive technology should inform better design of products for children—the time spent with children will have deeper, less tangible benefits—an understanding of the child’s world, a moment to lapse back into a space long departed, and a gentle, much-needed, confirmati on that humanity still has possibilities.
ABOU T THE AUTHORS
Dr. Janet Read is director of the
Child Computer Interaction Group at
the University of Central Lancashire
(UCLan) in the UK and Dr. Panos
Markopoulos is an associate profes-
sor at the Technical University of Eindhoven in the Netherlands. Dr.
Read has a first degree in mathematics from the University of
Manchester and a Ph.D. in child computer interaction from UCLan.
Dr. Markopoulos studied undergraduate computer science at the National Technical University of Athens and specialized in human-computer interaction at Queen Mary University of London, where he also did his doctorate in formal methods in human-computer interaction. Both authors have been heavily involved in the Interaction Design and Children conference series. Dr. Markopoulos co-chaired with Mathilde Bekker the first Interaction Design and Children Conference in 2002 and Dr. Read co-chaired the follow up event in 2003. Together the authors have presented several tutorials and workshops on child computer interaction and interaction design for children and have recently, with Stuart MacFarlane and Johanna Hoysniemi, written a specialist book entitled Evaluating Children’s Interactive Products: Principles and Practices for Interaction Designers published by Morgan Kaufmann.
November + December 2008
DOI: 10.1145/1409040.1409047
© 2008 ACM 1072-5220/08/1100 $5.00
References:
Archives