distribution in their own classes (“very often” to “never” on a
Likert scale), and how they would categorize the distribution
(normal, bimodal, multimodal, uniform, and others). We
randomly assigned participants to one of two treatments:
Treatment 0: participants were asked whether they
agreed that CS ability is innate, then asked to categorize
the distributions, and were not being primed to think
Treatment 1: participants were primed to think about the
common-held belief about CS grade distributions, before
they saw the distributions; after that, we asked whether they
agreed that CS ability is innate.
The survey’s five pages are described in Table 1. For each
question, we created a shorthand label, shown in sans-serif,
for use in our analysis.
Because so many of the potential participants were our
colleagues, we deliberately did not collect names and identify information about participants. We did not want to
know who was or was not a participant, nor how they
responded to the survey.
As a courtesy, we offered participants the option of
having their email recorded on a separate platform if
they wanted us to follow up with them about the results
of the study.
We did not look at this email list until after our analysis
4. 2. Participants
We recruited 60 CS instructors, mostly from the SIGCSE
members’ list. Some participants were recruited from other
online CS education communities, and some were recruited
at ICER 2015. Fifty-three participants completed every ques-
tion on the survey; twenty-eight were in Treatment 0 (the
nonprimed group), and twenty-five were in Treatment 1 (the
primed group). The participants who had provided their
emails for follow-up purposes were debriefed. As fewer than
half of the participants had provided their email, we posted
open debriefing statements to the online communities
where we had recruited participants.
4. 3. Results
For each participant, we computed a value we call “
seeing-bimodality,” which is how many of the six distributions the
participant had categorized as bimodal or multimodal. In
our data, seeing-bimodality ranged from 0 to 5.
Regression on seeing-bimodality. We wanted to see if
seeing-bimodality could be predicted by participants’ responses
to our questions. The regression we performed was to model
seeing-bimodality as a function of innately-predisposed,
all- succeed, look-histo, and look-letter (shorthand names from
When visualizing the results, we noticed that the relationship between seeing-bimodality and the Likert questions
varied between the two treatments. As a nonparametric
equivalent of ANCOVA, we performed an ordinal logistic
regression on the two treatments separately using the
polr function from R’s MASS library, and then used the
Anovafunction from the car package to compare the two.
This allowed us not only to test whether there were relationships between seeing-bimodality and the Likert questions,
but to see if these relationships were different for the two
treatments. This approach required computing 28 p values.
To reduce the chance of false positives from using multiple
statistical tests, we applied a Šidák correction, which
reduced our α level to 0.002 for this section of our analysis.
In both our regressions on Treatment 0 and on Treatment 1,
we found a significant relationship between seeing- bimodality
and participants’ responses to the questions related to innate
ability (all-succeed and innatelypredisposed).d
We then looked to see if this relationship was stronger in
one treatment than the other. In both questions about
innate ability, the effect was significantly stronger in the
treatment where subjects were primed to think about CS
grades being bimodal, as shown in Table 2.
Both regressions also revealed a statistically significant
relationship between seeing-bimodality and how often participants reported looking at histograms of their grades (
look-histo). This relationship was not statistically significantly
different between the two treatment groups.
Perhaps unsurprisingly, there was a strong negative
Table 2. Results of the Anova of the regressions on the two
treatments; that is, does the relationship between a given factor and
seeing-bimodality differ between the two treatments?
LR Chisq Df Signif?
d Regression tables are provided in the original ICER publication, and are
omitted due to page limitations.
Table 1. The pages of the survey.
1. Questions about how large their typical class was (“class-size”) and
how long they had been teaching (“years-experience”).
2. A priming question: ‘It is a commonly held belief that CS grade
distributions are bimodal. Do you find this to be the case in your
3. Questions on how often they look at their grade distributions:
• ‘When teaching, how often do you look at histograms of your
students’ grades? (This applies both to term work and final grades.)’
• ‘How often do you look at how many students fall into each letter
category (A, B, etc.)? (This applies both to term work and final
4. Six histograms, all generated with GNU R’s rnorm, shown in Figure 1.
For each histogram, we asked two questions:
• ‘How often do you see the shape of [this distribution] in your
• ‘What sort of distribution would you describe [this distribution] as?’
5. Likert-style questions on innate ability ( 5 points, Strongly Agree to
• Nearly everyone is capable of succeeding in computer science if they
work at it. (“all-succeed”)
• Some students are innately predisposed to do better at CS than
Pages 2 and 5 were swapped for a random half of the participants. We chose the
all-succeed question because it had been used previously in the literature.