Doi: 10.1145/1897852.1897875
The Informatics Philharmonic
By Christopher Raphael
abstract
A system for musical accompaniment is presented in which
a computer-driven orchestra follows and learns from a soloist in a concerto-like setting. The system is decomposed into
three modules: The first computes a real-time score match
using a hidden Markov model; the second generates the
output audio by phase-vocoding a preexisting audio recording; the third provides a link between these two, by predicting future timing evolution using a Kalman Filter–like
model. Several examples are presented showing the system
in action in diverse musical settings. Connections with
machine learning are highlighted, showing current weaknesses and new possible directions.
1. MusicaL accoMPaniMent systeMs
Musical accompaniment systems are computer programs
that serve as musical partners for live musicians, usually
playing a supporting role for music centering around the live
player. The types of possible interaction between live player
and computer are widely varied. Some approaches create
sound by processing the musician’s audio, often driven by
analysis of the audio content itself, perhaps distorting, echoing, harmonizing, or commenting on the soloist’s audio in
largely predefined ways. 8, 12 Other orientations are directed
toward improvisatory music, such as jazz, in which the computer follows the outline of a score, perhaps even composing
its own musical part “on the fly,” 3 or evolving as a “call and
response” in which the computer and human alternate the
lead role. 6, 9 Our focus here is on a third approach that models the traditional “classical” concerto-type setting in which
the computer performs a precomposed musical part in a
way that follows a live soloist. 2, 4, 11 This categorization is only
meant to summarize some past work, while acknowledging
that there is considerable room for blending these scenarios, or working entirely outside this realm of possibilities.
The motivation for the concerto version of the problem
is strikingly evident in the Jacobs School of Music (JSoM) at
Indiana University, where most of our recent experiments
have been performed. For example, the JSoM contains about
200 student pianists, for whom the concerto literature is
central to their daily practice and aspirations. However, in
the JSoM, the regular orchestras perform only two piano
concerti each year using student soloists, thus ensuring that
most of these aspiring pianists will never perform as orchestral soloist while at IU. We believe this is truly unfortunate
since nearly all of these students have the necessary technical skills and musical depth to greatly benefit from the
concerto experience. Our work in musical accompaniment
systems strives to bring this rewarding experience to the
music students, amateurs, and many others who would like
to play as orchestral soloist, though, for whatever reason, do
not have the opportunity.
Even within the realm of classical music, there are a
number of ways to further subdivide the accompaniment
problem, requiring substantially different approaches.
The JSoM is home to a large string pedagogy program
beginning with students at 5 years of age. Students in this
program play solo pieces with piano even in their first year.
When accompanying these early-stage musicians, the
pianist’s role is not simply to follow the young soloist, but
to teach as well, by modeling good rhythm, steady tempo
where appropriate, while introducing musical ideas. In a
sense, this is the hardest of all classical music accompaniment problems, since the accompanist must be expected
to know more than the soloist, thus dictating when the
accompanist should follow, as well as when and how to
lead. A coarse approximation to this accompanist role
provides a rather rigid accompaniment that is not overly
responsive to the soloist’s interpretation (or errors)—there
are several commercial programs that take this approach.
The more sophisticated view of the pedagogical music
system—one that follows and leads as appropriate—is
almost completely untouched, possibly due to the difficulty of modeling the objectives. However, we see this area
as fertile for lasting research contributions and hope that
we, and others, will be able to contribute to this cause.
An entirely different scenario deals with music that
evolves largely without any traditional sense of rhythmic
flow, such as in some compositions of Penderecki, Xenakis,
Boulez, Cage, and Stockhausen, to name some of the more
famous examples. Such music is often notated in terms of
seconds, rather than beats or measures, to emphasize the
irrelevance of regular pulse. For works of this type involving
soloist and accompaniment, the score can indicate points
of synchronicity, or time relations, between various points
in the solo and accompaniment parts. If the approach
is based solely on audio, a natural strategy is simply to wait
until various solo events are detected, and then to respond
to these events. This is the approach taken by the IRCAM
score follower, with some success in a variety of pieces of
this type. 2
A third scenario, which includes our system, treats works
for soloist and accompaniment having a continuing musical
pulse, including the overwhelming majority of “common
practice” art music. This music is the primary focus of most
of our performance-oriented music students at the JSoM,
and is the music where our accompaniment system is most
at home. Music containing a regular, though not rigid, pulse
requires close synchronization between the solo and accom-
panying parts, as the overall result suffers greatly as this
The original version of this chapter is entitled “Music
Plus One and Machine Learning” and was published in
Proceedings of the International Conference on Machine
Learning, Haifa, 2010.
