Elliott Tew’s results make a strong
case that pseudocode can be used effectively in a language-independent
test for CS1 knowledge and that her
test, in particular, is testing the same
kinds of things that CS1 teachers are
looking for on their final exams. But
the relevant finding is that the majority of her 952 test-takers failed both of
her exams, based on a small subset of
what anyone teaches in CS1. The average score on the pseudocode exam was
33.78%, and 48.61% on the “native” language exam. 10
These four studies4, 5, 8, 10 paint a picture of a nearly three-decades-long
failure to teach CS1 to a level that we
would expect. They span decades, a
variety of languages, and different
teaching methodologies, yet the outcomes are surprisingly similar. Certainly, the majority of students do
pass CS1, but maybe they shouldn’t
be passing. Each of these four efforts
to objectively measure student performance in CS1 has ended with the majority of students falling short of what
we might reasonably considerable
passable performance. The last three
studies, 4, 5, 10 in particular, have each
attempted to define a smaller and
smaller subset of what we might consider to be CS1 knowledge. Yet performance is still dismal. We have not yet
found a small enough definition of
CS1 learning outcomes such that the
majority of students achieve them.
There are a lot of possible explanations for why students perform so
badly on these measures. These studies may be flawed. Perhaps students
are entering the courses unprepared
for CS1. Perhaps our expectations for
CS1 are simply too high, and we should
not actually expect students to achieve
those learning objectives after only a
single course. Perhaps we just teach
CS1 badly. I argue that, regardless of
explanation, these four studies set us
up for success.
from science to engineering
In 1985, Halloun and Hestenes published a careful study of their use of
the Mechanics Diagnostics Test, later
updated as the Force Concept Inventory. 2 The Force Concept Inventory (FCI)
gave physics education researchers a
valid and reliable yardstick by which
to compare different approaches to
We need to develop
better ways of
science, like the
teaching physics knowledge about
force. The traditional approach was
clearly failing. Hestenes reported that
while “nearly 80% of the students
could state Newton’s Third Law at
the beginning of the course…FCI data
showed that less than 15% of them ful-
ly understood it at the end.”
FCI as a yardstick was the result of
physics education research as science.
Scientists define phenomena and de-
velop instruments for measuring those
phenomena. Like computer scientists,
education researchers are both scien-
tists and engineers. We not only aim
to define and measure learning—we
develop methods for changing and im-
Physics education researchers defined a set of new methods for teaching physics called
interactive-engagement methods. These methods move
away from traditional lecture, and
instead focus on engaging students
in working with the physics content.
In a study with a stunning 6,000-plus
methods were clearly established to
be superior to traditional methods for
teaching physics. 1 Once the yardstick
was created, it was possible to engineer new ways of teaching and compare them with the yardstick.
The demands of the “Rising Above
the Gathering Storm” and PCAST re-
ports call on computing education re-
searchers to be engineers, informed
by science. These four studies estab-
lish a set of measures for CS1 knowl-
edge. There are likely flaws in these
measures. More and better measures
can and should be developed. There is
much that we do not know about how
students learn introductory comput-
ing. There is plenty of need for more
1. Hake, R.R. Interactive-engagement vs. traditional
methods: A six-thousand-student survey of mechanics
test data for instructory physics courses. American
Journal of Physics 66 (1998), 64–74.
2. Hestenes, D. et al. Force Concept Inventory. Physics
Teacher 30 (1992), 141–158.
3. Lemann, N. Schoolwork: The overblown crisis
in American education. The New Yorker (Sept.
27, 2010); http://www.newyorker.com/talk/
4. Lister, R. et al. A multi-national study of reading and
tracing skills in novice programmers. Working group
reports from ITiCSE Conference on Innovation and
Technology in Computer Science Education. ACM,
Leeds, U. K., 2004, 119–150.
5. McCracken, M. et al. A multi-national, multi-institutional study of assessment of programming
skills of first-year CS students. SIGCSE Bulletin 33, 4
6. Prepare and Inspire: K– 12 Education in Science,
Technology, Engineering, and Math (STEM) for
America’s Future. Executive Office of the President,
Washington, D. C., 2010.
7. Rampell, C. Once a dynamo, the tech sector is slow to
hire. The New York Times (Sept. 7, 2010); http://www.
8. Rising Above the Gathering Storm, Revisited: Rapidly
Approaching Category 5. National Academies Press,
Washington, D. C., 2010.
9. Soloway, E. Cognitive strategies and looping
constructs: An empirical study. Commun. ACM 26, 11
(Nov. 1983), 853–860.
10. Tew, A.E. Assessing fundamental introductory
computing concept knowledge in a language
independent manner. Ph.D. in Computer Science:
School of Interactive Computing. Georgia Institute of
Technology. Atlanta, GA (2010).
Mark Guzdial ( email@example.com) is a professor
in the College of Computing at Georgia Institute of
Technology in Atlanta, GA.
Copyright held by author.