focus groups. For very limited domains, it may be possible to automate data gathering of this kind. However, this is not qualitative research. Putting a human in the loop does not necessarily make research qualitative; telephone surveys that ask interviewers to follow pro forma scripts in effect turn humans into data-gathering robots. Qualitative interview studies— whether conducted by phone or online—involve sensitive, active listening. That is, listening carefully to nuance and inflection, offering follow-up questions, working to build trust, and being candid. Qualitative interviews are carefully guided conversations.

2. Reduce time spent on the project. Some time in the field is better than no time. But sloppy work is expensive, and to really have effective short-term qualitative investigations, you need to triangulate—think about your questions and use all the methods you can to hone in on the core question that needs answering. In the mid-1990s, Tony Salvador and Michael Mateas coined the phrase “garage ethnography” to talk about quick-turnaround field investigations. They understood that there are a lot of different methods for researching and understanding people and their activities: Some time in the field was better than no time. Moreover, one should be circumspect about the results and be clear with caveats as well as talking points. That is, they pointed to the limitations as well as the benefits of short investigations in deriving thick and thin, detailed and abstract, descriptions. These

were researchers who knew how to go from quick to long studies, to triangulate methods, and to recognize when the data they collected did not deliver results.

Many companies are really good at “quick and dirty” qualitative studies. The good can be separated from the bad by a few questions that should have well-thought-out answers: how/why did you select the location(s) and activity setting(s), how/why did you select your informant(s), what data did you gather and how, what analyses are you doing, what has surprised you…. There are more questions to pose, of course, but these are the basics.

3. Simplify data analysis, automate the report generation. One way to cut costs is to eliminate the analysis phase, and/or have data crunchers—human or machine—simply summarize the results. In a recent analysis of available services, several of the sites I reviewed brag about their automated analysis techniques. I am deeply suspicious of such claims. Data analysis should be an ongoing weaving of themes. Those themes should contribute to a broader explanation of theoretical or practical import and guide a summary report for discussion. A focus on discovery is important— verifying established ideas should not be the only goal. Qualitative studies are reflective and reflexive, analysis and synthesis involve finding patterns, and really good research requires time for going back and revisiting one’s assumptions. If someone is not carefully analyzing and thoughtfully synthesising data, he or she is doing you a disservice.

Perhaps more controversially, I would caution against research plans that do not entertain possible revision in response to ongoing data analysis. In a qualitative study, research activities take shape gradually, as meaning unfolds through ongoing, concurrent data collection, and analysis. Like recent perspectives in software development, research processes should be agile. Rather than signifying clarity, professionalism, or certainty, research that is executed precisely to prespecified procedures may signify rigidity and lack of sensitivity to the issues at hand.

As the consumer of services, you should ask to see previous reports that are available for public release. Beware of slide decks filled with data that do not make sense to you; ask what analysis methods were used; ask about concurrent analysis processes; and ask about interim research process reviews. If your analysts have never heard of field memos or content analysis techniques and cannot tell you how they analyzed the data they gathered for you, and you don’t get to see any of the raw data (anonymized, if appropriate), be wary. Good researchers leave you with the knowledge of how to replicate the study, not just the study results. Finally, be very suspicious of anyone who tells you how many hours of data they gathered and how many lines of transcription they analyzed and waits for you to gasp with excitement at the impressive figures cited. This is akin to judging a computer program by the number of lines of code.