ably well. Otherwise, the player will become distracted
by the poor ensemble and not be able to demonstrate
what he or she wants to hear. Thus there must be a
neutral setting of parameters that allows the system to
perform reasonably well “out of the box.”
4. 1. the timing model
We first consider a timing model for a single musical part.
Our model is expressed in terms of two hidden sequences,
{tn} and {sn} where tn is the time, in seconds, of nth note
onset and sn is the tempo, in seconds per beat, for the nth
note. These sequences evolve according to the model
sn + 1 = sn + sn
( 3)
tn + 1 = tn + lnsn + tn
( 4)
where ln is the length of the nth event, in beats.
With the “update” variables, {sn} and {tn}, set to 0,
this model gives a literal and robotic musical performance with each inter-onset-interval, tn + 1 − tn, consuming
an amount of time proportional to its length in beats, ln.
The introduction of the update variables allows time-varying tempo through the {sn}, and elongation or compression of note lengths with the {tn}. We further assume
that the {(sn, tn)t} are independent with (sn,tn)t ∼ N(mn, Gn),
n = 1, 2,..., and (s0, t0)t N(m0, G0), thus leading to a joint
Gaussian model on all model variables. The rhythmic
interpretation embodied by the model is expressed in
terms of the {mn, Gn} parameters. In this regard, the {mn}
vectors represent the tendencies of the performance—
where the player tends to speed up (sn < 0), slow down
(sn > 0), and stretch (tn > 0), while the {Gn} matrices capture the repeatability of these tendencies.
It is simplest to think of Equations 3 and 4 as a timing
model for single musical part. However, it is just as reasonable to view these equations as a timing model for the
composite rhythm of the solo and orchestra. That is, consider the
situation, depicted in Figure 3, in which the solo, orchestra,
and composite rhythms have the following musical times
(in beats):
solo 0 1/3
accomp 0 1/2
comp. 0 1/3 1/2
2/3 1 4/3
5/3 2
1 3/2
2
2/3 2 4/3 3/2 5/3
2
The {ln} for the composite would be found by simply taking
the differences of rational numbers forming the composite
rhythm: l1 = 1/3, l2 = 1/6, etc. In what follows, we regard
Equations 3 and 4 as a model for this composite rhythm of
the solo and orchestra parts.
The observable variables in this model are the solo
note onset estimates produced by Listen and the known
note onsets of the orchestra (our system constructs these
during the performance). Suppose that n indexes the
events in the composite rhythm having associated solo
notes, estimated by the {ˆtn}. Additionally, suppose that n¢
indexes the events having associated orchestra notes with
33
figure 3. top: two musical parts generate a composite rhythm
when superimposed. Bot: the resulting graphical model arising
from the composite rhythm.
Solo
Accomp
Composite
Listen
Updates
Accomp
onset times, {on¢}. We model
ˆtn = tn + n
on¢ = tn¢ + dn¢
4. 2. the model in action
With the model in place we are now ready for real-time accompaniment. For our first rehearsal we initialize the model
so that mn = 0 for all n. This assumption in no way precludes
our system from correctly interpreting and following tempo
changes or other rhythmic nuances of the soloist. Rather, it
states that, whatever we have seen so far in a performance, we
expect future timing to evolve according to the current tempo.
In real-time accompaniment, our system is concerned only
with scheduling the currently pending orchestra note time,
on¢. The time of this note is initially scheduled when we play
the previous orchestra note, on¢− 1. At this point we compute the
new mean of on¢, conditioning on on¢− 1 and whatever other variables have been observed, and schedule on¢ accordingly. While
we wait for the currently scheduled time to occur, the Listen
module may detect various solo events, ˆtn. When this happens
we recompute the mean of on¢, conditioning on this new information. Sooner or later the actual clock time will catch up to
the currently scheduled time of the n¢th event, at which point
the orchestra note is played. Thus an orchestra note may be
rescheduled many times before it is actually played.
A particularly instructive example involves a run of many