ably well. Otherwise, the player will become distracted by the poor ensemble and not be able to demonstrate what he or she wants to hear. Thus there must be a neutral setting of parameters that allows the system to perform reasonably well “out of the box.”

4. 1. the timing model

We first consider a timing model for a single musical part. Our model is expressed in terms of two hidden sequences, {tn} and {sn} where tn is the time, in seconds, of nth note onset and sn is the tempo, in seconds per beat, for the nth note. These sequences evolve according to the model

sn + 1 = sn + sn

( 3)

tn + 1 = tn + lnsn + tn

( 4)

where ln is the length of the nth event, in beats.

With the “update” variables, {sn} and {tn}, set to 0, this model gives a literal and robotic musical performance with each inter-onset-interval, tn + 1 − tn, consuming an amount of time proportional to its length in beats, ln. The introduction of the update variables allows time-varying tempo through the {sn}, and elongation or compression of note lengths with the {tn}. We further assume that the {(sn, tn)t} are independent with (sn,tn)t ∼ N(mn, Gn), n = 1, 2,..., and (s0, t0)t N(m0, G0), thus leading to a joint Gaussian model on all model variables. The rhythmic interpretation embodied by the model is expressed in terms of the {mn, Gn} parameters. In this regard, the {mn} vectors represent the tendencies of the performance— where the player tends to speed up (sn < 0), slow down (sn > 0), and stretch (tn > 0), while the {Gn} matrices capture the repeatability of these tendencies.

It is simplest to think of Equations 3 and 4 as a timing model for single musical part. However, it is just as reasonable to view these equations as a timing model for the composite rhythm of the solo and orchestra. That is, consider the situation, depicted in Figure 3, in which the solo, orchestra, and composite rhythms have the following musical times (in beats):

solo 0 1/3

accomp 0 1/2

comp. 0 1/3 1/2

2/3 1 4/3

5/3 2

1 3/2

2

2/3 2 4/3 3/2 5/3

2

The {ln} for the composite would be found by simply taking the differences of rational numbers forming the composite rhythm: l1 = 1/3, l2 = 1/6, etc. In what follows, we regard Equations 3 and 4 as a model for this composite rhythm of the solo and orchestra parts.

The observable variables in this model are the solo note onset estimates produced by Listen and the known note onsets of the orchestra (our system constructs these during the performance). Suppose that n indexes the events in the composite rhythm having associated solo notes, estimated by the {ˆtn}. Additionally, suppose that n¢ indexes the events having associated orchestra notes with

33

figure 3. top: two musical parts generate a composite rhythm when superimposed. Bot: the resulting graphical model arising from the composite rhythm.

Solo Accomp Composite Listen Updates Accomp

onset times, {on¢}. We model
ˆtn = tn + n
on¢ = tn¢ + dn¢

4. 2. the model in action

With the model in place we are now ready for real-time accompaniment. For our first rehearsal we initialize the model so that mn = 0 for all n. This assumption in no way precludes our system from correctly interpreting and following tempo changes or other rhythmic nuances of the soloist. Rather, it states that, whatever we have seen so far in a performance, we expect future timing to evolve according to the current tempo.

In real-time accompaniment, our system is concerned only with scheduling the currently pending orchestra note time, on¢. The time of this note is initially scheduled when we play the previous orchestra note, on¢− 1. At this point we compute the new mean of on¢, conditioning on on¢− 1 and whatever other variables have been observed, and schedule on¢ accordingly. While we wait for the currently scheduled time to occur, the Listen module may detect various solo events, ˆtn. When this happens we recompute the mean of on¢, conditioning on this new information. Sooner or later the actual clock time will catch up to the currently scheduled time of the n¢th event, at which point the orchestra note is played. Thus an orchestra note may be rescheduled many times before it is actually played.

A particularly instructive example involves a run of many