Communications - February 2011

Elliott Tew’s results make a strong case that pseudocode can be used effectively in a language-independent test for CS1 knowledge and that her test, in particular, is testing the same kinds of things that CS1 teachers are looking for on their final exams. But the relevant finding is that the majority of her 952 test-takers failed both of her exams, based on a small subset of what anyone teaches in CS1. The average score on the pseudocode exam was 33.78%, and 48.61% on the “native” language exam. 10

These four studies4, 5, 8, 10 paint a picture of a nearly three-decades-long failure to teach CS1 to a level that we would expect. They span decades, a variety of languages, and different teaching methodologies, yet the outcomes are surprisingly similar. Certainly, the majority of students do pass CS1, but maybe they shouldn’t be passing. Each of these four efforts to objectively measure student performance in CS1 has ended with the majority of students falling short of what we might reasonably considerable passable performance. The last three studies, 4, 5, 10 in particular, have each attempted to define a smaller and smaller subset of what we might consider to be CS1 knowledge. Yet performance is still dismal. We have not yet found a small enough definition of CS1 learning outcomes such that the majority of students achieve them.

There are a lot of possible explanations for why students perform so badly on these measures. These studies may be flawed. Perhaps students are entering the courses unprepared for CS1. Perhaps our expectations for CS1 are simply too high, and we should not actually expect students to achieve those learning objectives after only a single course. Perhaps we just teach CS1 badly. I argue that, regardless of explanation, these four studies set us up for success.

from science to engineering In 1985, Halloun and Hestenes published a careful study of their use of the Mechanics Diagnostics Test, later updated as the Force Concept Inventory. 2 The Force Concept Inventory (FCI) gave physics education researchers a valid and reliable yardstick by which to compare different approaches to

We need to develop
better ways of
teaching computer
science, like the
physics educators’
interactive-
engagement
methods.

teaching physics knowledge about
force. The traditional approach was
clearly failing. Hestenes reported that
while “nearly 80% of the students
could state Newton’s Third Law at
the beginning of the course…FCI data
showed that less than 15% of them ful-
ly understood it at the end.”
FCI as a yardstick was the result of
physics education research as science.
Scientists define phenomena and de-
velop instruments for measuring those
phenomena. Like computer scientists,
education researchers are both scien-
tists and engineers. We not only aim
to define and measure learning—we
develop methods for changing and im-
proving it.

Physics education researchers defined a set of new methods for teaching physics called interactive-engagement methods. These methods move away from traditional lecture, and instead focus on engaging students in working with the physics content. In a study with a stunning 6,000-plus participants interactive-engagement methods were clearly established to be superior to traditional methods for teaching physics. 1 Once the yardstick was created, it was possible to engineer new ways of teaching and compare them with the yardstick.

The demands of the “Rising Above
the Gathering Storm” and PCAST re-
ports call on computing education re-
searchers to be engineers, informed
by science. These four studies estab-
lish a set of measures for CS1 knowl-
edge. There are likely flaws in these
measures. More and better measures
can and should be developed. There is
much that we do not know about how
students learn introductory comput-
ing. There is plenty of need for more
science.

References

1. Hake, R.R. Interactive-engagement vs. traditional methods: A six-thousand-student survey of mechanics test data for instructory physics courses. American Journal of Physics 66 (1998), 64–74.

2. Hestenes, D. et al. Force Concept Inventory. Physics Teacher 30 (1992), 141–158.

3. Lemann, N. Schoolwork: The overblown crisis in American education. The New Yorker (Sept.

27, 2010); http://www.newyorker.com/talk/ comment/2010/09/27/100927taco_talk_ lemann#ixzz10GexVQJU.

4. Lister, R. et al. A multi-national study of reading and tracing skills in novice programmers. Working group reports from ITiCSE Conference on Innovation and Technology in Computer Science Education. ACM, Leeds, U. K., 2004, 119–150.

5. McCracken, M. et al. A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. SIGCSE Bulletin 33, 4 (2001), 125–180.

6. Prepare and Inspire: K– 12 Education in Science, Technology, Engineering, and Math (STEM) for America’s Future. Executive Office of the President, Washington, D. C., 2010.

7. Rampell, C. Once a dynamo, the tech sector is slow to hire. The New York Times (Sept. 7, 2010); http://www. nytimes.com/2010/09/07/business/economy/07jobs. html.

8. Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5. National Academies Press, Washington, D. C., 2010.

9. Soloway, E. Cognitive strategies and looping constructs: An empirical study. Commun. ACM 26, 11 (Nov. 1983), 853–860.

10. Tew, A.E. Assessing fundamental introductory computing concept knowledge in a language independent manner. Ph.D. in Computer Science: School of Interactive Computing. Georgia Institute of Technology. Atlanta, GA (2010).