Crossroads - Summer 2014

cisions, describing why we did something, not only what we have done. This is not easy, as it often requires making explicit the elements practitioners use intuitively. But such documentation is extremely valuable, especially for others, to understand why some things worked well or not.

Generalization. Knowledge obtained from practice can be viewed as a generalization of experiences. In a normal design effort the primary goal is to create a successful product, and lessons learned are restricted to the particular design and the people involved in it. To be useful to others, some effort has to be invested in generalizing these lessons.

Generalization enables correlating different experiences, which other wise may look too specific. In the process of generalization, practitioners need to expand their focus beyond the current design situation, viewing the design problem, solutions, and processes as instances of more general classes. However it is also important not to “over-generalize.” Practice usually cannot provide us with insights to develop “grand” and universal theories. Rather, emphasis should be between narrow truths specific to some situation, and broader knowledge covering several similar situations.

Evaluation. Evaluation is a necessary part of any learning process. This is especially true for domains, such as software engineering or human-computer interaction, where we have to deal with complex human and social issues for which we do not have strong theories, models, and laws. By evaluation I do not necessarily mean formal evaluation activities conducted in “laboratories” (even though these may be sometimes used). I see evaluation as a systematic effort to get feedback on our findings and quality of our work that is not based on our intuition.

Even simple techniques, such as peer reviews may be incredibly efficient in identifying shortfalls in our problem analysis, the solution, and the design procedure. In order to enable such a process, practitioners need to be prepared to make their reasoning explicit, public, and open to critical reflection and discussion. The key is to make your intuitive decisions more explicit and “vulnerable” to the critique of others and empirical findings.

Iterations. Lastly, to maximize learning, all of the previous elements should be applied in a number of iteration. While a single event can have an impact, it usually takes many events to extract general features and generate rules from experience. In everyday work, you should try to combine elements of preparation, actual practical work, and evaluation.

Learning should not stop at the
end of a project. New insights and
broader generalizations will often oc-
cur through retrospective analyses
of lessons learned and data collected
through a whole project. Brooks, for
example, spent several years analyzing
and reflecting on the lessons learned
in design of OS/360, producing the
influential The Mythical Man-Month
[ 5]. These macro-iterations of retro-
spective analyses can also happen on
a broader scale, covering several proj-
ects from different contributors. For
example, in special issues of journals,
editors often spend some time sum-
marizing and generalizing findings
from individual articles.

CONCLUSION

Doing research in practice is valuable both for the practice and for the research community. But for you as a practitioner, doing research in practice may be a competitive advantage. You may bring to practice original ideas and results of cutting edge research, and you may bring to the research community lessons researchers from academia do not have the opportunity to obtain.

To read the full version of this article, please visit http://xrds.acm.org/article. cfm?aid=2611810.

References

[ 1] Obrenović, Z. Design-based Research: What we learn when we engage in design of interactive systems.

Interactions 18, 5 (2011), 56-59.

[ 2] Obrenović, Z. Rethinking HCI education: teaching interactive computing concepts based on the experiential learning paradigm. Interactions 19, 3

(2012), 66-70.

[ 3] Obrenović, Z. Research and Practice: The curious case of ‘small’ researchers-practitioners. Communications of the ACM 56, 9 (2013), 38-40.

[ 4] Simon, H. A. The Sciences of the Artificial. MI T Press, Cambridge, 1996.

[ 5] Brooks, F.P. The Mythical Man-Month. Addison-Wesley Professional, 1995.

[ 6] Brooks, F.P. The Design of Design: Essays from a Computer Scientist. Addison- Wesley Professional,

2010.

Biography

Željko Obrenović is a technical consultant with the Software Improvement Group in Amsterdam, The Netherlands. Before joining SIG he worked as a researcher and best practices consultant at Backbase, an assistant professor at the Technical University in Eindhoven, and as a researcher at C WI in Amsterdam. In his work, he aims at bridging software design research and practice, trying to get best of both worlds.