N. Why people hate your app: Making sense of user
feedback in a mobile app store. In Proceedings of
the 19th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (Chicago, IL,
Aug. 11–14). ACM Press, New York, 2013, 1276–1284.
6. Galvis Carreño, L.V. and Winbladh, K. Analysis of user
comments: An approach for software requirements
evolution. In Proceedings of the 2013 International
Conference on Software Engineering (San Francisco,
CA, May 18–26). IEEE Press, Piscataway, NJ, 2013,
7. Google Analytics; http://www.google.ca/analytics/mobile/
8. Guzman, E. and Maalej, W. How do users like this
feature? A fine-grained sentiment analysis of app
reviews. In Proceedings of the 22nd International
Requirements Engineering Conference (Karlskrona,
Sweden, Aug. 25–29). IEEE Press, Piscataway, NJ,
9. Harman, M., Jia, Y., and Zhang, Y. App store mining
and analysis: MSR for app stores. In Proceedings of
the Ninth Working Conference on Mining Soft ware
Repositories (Zurich, Switzerland, June 2–3).
Piscataway, NJ, 2012.
10. Harrell, F.E. Regression Modeling Strategies: With
Applications to Linear Models, Logistic Regression, and
Survival Analysis. Springer, New York, 2001.
11. Hoon, L., Vasa, R., Schneider, J.-G., Grundy, J. et al. An
Analysis of the Mobile App Review Landscape: Trends and
Implications. Technical Report. Swinburne University of
Technology, Faculty of Information and Communication
Technologies, Melbourne, Australia, 2013.
12. Iacob, C. and Harrison, R. Retrieving and analyzing
mobile apps feature requests from online reviews. In
Proceedings of the 10th International Workshop on
Mining Software Repositories (San Francisco, CA, May
18–19). IEEE Press, Piscataway, NJ, 2013, 41–44.
13. Johns, T. Replying to User Reviews on Google Play.
Android Developers Blog, June 21, 2012; http://
14. Khalid, H., Shihab, E., Nagappan, M., and Hassan, A.
What do mobile app users complain about? IEEE
Software 32, 3 (May-June 2015), 70–77.
15. Kim, H.-W., Lee, H.L., and Son, J.E. An exploratory
study on the determinants of smartphone app
purchase. In Proceedings of the 11th International
DSI Decision Sciences Institute and 16th APDSI Asia
Pacific Region of Decision Sciences Institute Joint
Meeting (Taipei, Taiwan, July 12–16, 2011).
16. Lim, S. L., Bentley, P. J., Kanakam, N., Ishikawa, F.,
and Honiden, S. Investigating country differences
in mobile app user behavior and challenges for
software engineering. IEEE Transactions on Software
Engineering 41, 1 (Jan. 2015), 40–64.
17. Martin, W., Harman, M., Jia, Y., Sarro, F., and Zhang, Y.
The app-sampling problem for app store mining. In
Proceedings of the 12th Working Conference on Mining
Software Repositories (Florence, Italy, May 16–17).
IEEE Press, Piscataway, NJ, 2015.
18. Mudambi, S.M. and Schu, D. What makes a helpful
online review? A study of customer reviews on
Amazon.com. MIS Quarterly 34, 1 (2010), 185–200.
19. Pagano, D. and Bruegge, B. User involvement
in software evolution practice: A case study. In
Proceedings of the 2013 International Conference
on Software Engineering (San Francisco, May 18–26).
IEEE Press, Piscataway, NJ, 2013, 953–962.
20. Pagano, D. and Maalej, W. User feedback in the App
Store: An empirical study. In Proceedings of the
21st IEEE International Requirements Engineering
Conference (Rio de Janeiro, Brazil, July 15–19). IEEE,
Piscataway, NJ, 2013.
Stuart Mcilroy ( firstname.lastname@example.org) is a Ph.D.
student at Dalhousie University, Halifax, Canada.
Weiyi Shang ( email@example.com) is an assistant
professor and Concordia University Research Chair
in Ultra-Large-Scale Systems in the Department of
Computer Science and Software Engineering at Concordia
University, Montreal, Canada.
Nasir Ali ( firstname.lastname@example.org) is an assistant research
professor at the University of Memphis, Memphis, TN.
Ahmed E. Hassan ( email@example.com) is Canada
Research Chair in Software Analytics and NSERC/
BlackBerry Software Engineering Chair in the School of
Computing at Queen’s University, Kingston, Canada.
© 2017 ACM 0001-0782/17/11
dence of a spike in reviews following a
Implication. Greater effort examining
user reviews should follow a release in
order to improve app quality.
Pagano and Maalej20 reported that
the number of received reviews decreased over time after a release, suggesting releases contribute to new reviews. We observed the same kind of
correlation for the Google Play store.
Figure 4 outlines a boxplot of the median number of reviews for each studied
app across each of its releases, showing
a spike in reviews directly on and after
an app’s release day.
However, still not clear is if these
spikes were due to an app attracting
new users following its release or to current users becoming more inclined to
review the app. Looking closer at our
nomogram, we note that many releases
(more than 20) for an app has as much
of an effect as an app with 10 million
downloads. Frequent releases thus ensure an app’s user base is more engaged
as it begins providing feedback.
A very small percentage of the top apps
we studied (0.19% of 10,713) have ever
received more than 500 reviews per day,
yet most studied apps received only a
few reviews per day. The number of received reviews for the studied apps did
not vary due to the category to which the
app belonged, varying instead based
on number of downloads and releases.
Some of our results highlight differences between the Google Play store and
the Apple App Store.
Additional studies are needed to
better understand the review dynamics
across both stores. Researchers should
thus examine whether other empirical findings hold across them. In particular, techniques designed to assist
mobile-app developers should be optimized for each store.
1. App Annie Analytics; http://www.appannie.com/app-store-analytics/
2. Applause; https://www.applause.com/testing/
3. Chandy, R. and Gu, H. Identifying spam in the iOS app
store. In Proceedings of the Second Joint WICO W/
AIR Web Workshop on Web Quality (Lyon, France, Apr.
16). ACM Press, New York, 2012, 56–59.
4. Chen, N., Lin, J., Hoi, S,C.H., Xiao, X., and Zhang, B.
AR-Miner: Mining informative reviews for developers
from the mobile app marketplace. In Proceedings
of the 36th International Conference on Software
Engineering (Hyderabad, India, May 31–June 7). ACM
Press, New York, 2014, 767–778.
5. Fu, B., Lin, J., Li, L., Faloutsos, C., Hong, J., and Sadeh,
ensure an app’s
user base is
as it begins