ACM’s
interactions
magazine explores
critical relationships
between experiences, people,
and technology, showcasing
emerging innovations and industry

CS1 was in a series of studies by Elliot Soloway and his colleagues at Yale University. They regularly used the same problem, called “The Rainfall Problem”: Write a program that repeatedly reads in positive integers, until it reads the integer 99999. After seeing 99999, it should print out the average. In one study, only 14% of students in Yale’s CS1 could solve this problem correctly. 9 The Rainfall Problem has been used under test conditions and as a take-home programming assignment, and is typically graded so that syntax errors don’t count, though adding a negative value or 99999 into the total is an automatic zero. Every study that I’ve seen (the latest in 2009) that has used the Rainfall Problem has found similar dismal performance, on a problem that seems amazingly simple.

Mike McCracken realized the problem with Soloway’s studies, or any similar study, could be that a single campus could get it wrong. Maybe Yale just taught CS1 badly. McCracken wanted to find problems that students might be having in general with CS1. He organized a multi-institutional, multi-national (MIMN) study, with student data on the same problem collected from four institutions in three different countries. 5 One place might get it wrong, use the “wrong” language, or use “objects-first” when they ought to do “objects-later” (or some other pedagogical trade-off). Studying a range of schools helps us to describe “ traditional” teaching of that subject, and student outcomes from that teaching. McCracken’s group asked students to evaluate arithmetic expressions where

 

http://www.acm.org/subscribe

computing education
research can help us
figure out where the
shortcomings are in
the u.s. education
system, and how to
address them.

the numbers and operations appeared in a text file (prefix, postfix, or infix— student’s choice). 215 CS1 students participated in the study. The average score was 21%. Many of the participants never got past the design part of the problem to write any code at all.

Raymond Lister thought that maybe McCracken’s study was asking students to do too much, in that they were designing solutions and implementing programs. He organized another MIMN study, this time where they asked students to read and trace code. 556 students from six institutions across seven countries completed 12 multiple-choice questions involving iteration and array manipulation. The average score was 60%; 23% of the students were only able to get four or fewer problems correct.

The most recent evaluation of CS1
is in the dissertation by Allison Elliott
Tew, whom I advised. 10 Elliott Tew has
been interested in how we can com-
pare performance between different
kinds of CS1, especially where the lan-
guage varies. She hypothesized that
students could take a test written in a
pseudocode, especially designed to be
easy to read, that would correlate well
with how students performed in what-
ever their “native” CS1 language was.
Before she created her test, though, she
had to define what we mean by “CS1.”
Since Elliott Tew wanted her test to
be usable in a variety of different kinds
of classes, she tried to define a small
subset of what different people saw as
“CS1 knowledge.” Elliott Tew looked at
popular CS1 textbooks to define the in-
tersection set of topics between those,
and used the ACM/IEEE curricular
volumes to identify only those topics
recommended for CS1. In the end, she
defined a very small subset of what any-
one teaches in CS1 as the “CS1 knowl-
edge” that she would test.

Elliott Tew created an exam with 27 questions in each of MATLAB, Python, Java, and her pseudocode. Each of her 952 subjects, from three institutions in two countries, completed two exams: One in her pseudocode, and one in their “native” language. She found that the correlation was very high between the pseudocode and the “native” language, and additionally, the correlation was very high between the pseudocode and the students’ final exam grade in CS1.