Communications of the ACM

vealed this interaction came about because the computer-generated blogs capturing conditions with more movement by kites (C2 and C3) were more informative than the human-written blogs for the same conditions and more informative than computer-generated blogs capturing constricted movement (C1) (p<0.0001 for each comparison).

To better understand these described effects, we compared the distribution of ratings obtained by each human writer (H1–H12) and the computer (Comp) in Figure 7. Only two of the blog writers (H3 and H10) were deemed to write more informative blogs than the computer, and both of them were considered less engaging and fluent than the computer-generated blogs. Likewise, H4, who wrote more fluent and engaging blogs than the computer, was rated rather low

Results

Evaluation against human-written blogs. Both sets of students showed an overall significant preference for the computer-generated blogs (238 trials vs. 153 trials in which human-written blogs were preferred; χ2 = 18. 5; p < 0.001), confirming hypothesis H1. However, a more complex pattern emerged (see Figure 5), with this preference being dependent on the type of kite movement covered in the blog—C1, C2, or C3—and the orientation of the course—ecology or technology.

Across the community ecology students, there was a strong preference for computer-generated blogs when they captured more extensive movement by the kites—round trips (C2) and movement between home ranges (C3)—while there was little difference in preference between the two blog types when kite movement was limited; that is, small movements within home ranges (C1). Digital society students showed an overall clear preference for the computer-generated blogs only when they described round trips (C2). Combined, our findings indicate Blogging Birds is particularly skilled at handling cases where the focal bird shows substantial movement. Average ratings for how fluent, engaging, and informative the blogs were (see Figure 6) showed the main perceived advantage of the computer-generated blogs pertains to their “ informativeness,” with smaller improvements visible for how engaging and fluent they were.

We ran a MANOVA, with informativeness, engagingness, and fluency as the dependent variables and blog type (computer or human), kite movement-pattern (C1, C2, or C3), student group (community ecology or digital society), and their interactions as fixed effects, and writer ID and evaluator ID as random effects. We found the following main effects and interactions at p<0.01: computer-written blogs were rated significantly higher (p<0.0001) than human-written blogs (confirming hypothesis H2); students in the digital society course gave higher ratings overall than students in community ecology (p<0.01); and there was interaction between blog type and movement pattern (p<0.0001), confirming hypothesis H3. Post-hoc analysis using the Tukey HSD test on the individual ANO-VAs with Bonferroni-correction re-

Figure 7. Computer-generated blogs (Comp) vs. human-written blogs (H1–H12).

H1H6H4H5H8H7H9H11H2H12

C ompH3H10

0

1

2

3

4

5

6

7

Me
an
Ra
t
in
g
(a) Informativeness

H 6

H1H5H8H7H3H9H2H10

C ompH4H12H11

(b) Engagingness Blog Writers

H 6H1H5H7H8H12H10H3H9

C ompH4H11H2

0

1

Pro por tionsofratings (c) Fluency

Likert Scale

7

6

5

4

3

2

1

Figure 8. Computer-generated blogs with ecological insights (full system) vs. computer-generated blogs describing movement patterns only (baseline system).