training for each and every new person our vendors in China and India hired to perform the testing for us. That training covered not only our model-based testing approach, but also some other aspects of the overall methodology.

BInDER: How long did it take for moderately competent developers who had never encountered model-based testing before to get to the point where they could actually be pretty productive?

KICILLoF: On average, I’d say that took close to a month.

BInDER: Once your testers were trained, how did your testing approach evolve? Did you run into any significant problems along the way?

GRIESKAMP: It proved to be a fairly smooth transition since we were just working with concepts that were part of the prototype we had already developed back at Microsoft Research. That said, it actually was just a prototype when this team took it over, so our main challenge was to stabilize the technology. You know how prototypes are—they crash and you end up having to do workarounds and so forth. We’ve had a development team working to improve the tool over the past three years, and thousands of fixes have come out of that.

Another potential issue had to do with something that often crops up in model-based testing: a state-explosion problem. Whenever you model—if you naively define some rules to update your state whenever certain con-

BoB BInDER
how did you
manage to convince
yourselves you
could take several
hundred test
developers who
had no experience
in this area and
teach them
a fairly esoteric
technique for
translating words
into rule systems?

ditions are met and then you just let the thing run—there’s a good chance you’re going to end up getting overrun by all those state updates. For example, when using this tool, if you call for an exploration, that should result in a visualization of the exploration graph that you can then inspect. If you’re not careful, however, you could end up with thousands and thousands of states the system will try to explore for you. There’s just no way you’re going to be able to visualize all of that.

Also, in order to see what’s actually going on, you need to have some way of pruning down the potential state space such that you can slice out those areas you know you’re going to need to test. That’s where one of our biggest challenges was: finding the right way to slice the model.

The idea here was to find the right slicing approach for any given problem, and the tool provides a lot of assistance for accomplishing that. It didn’t come as a surprise to us that this issue of finding the right way to slice the space would end up being a problem— we had expected that. We actually had already added some things to the tool to deal with that, which is probably one of the reasons the project has proved to be a success.

KICILLoF: The secret is to use test purposes as the criterion for slicing.

BInDER: With that being only a subset of all the behaviors you would be looking at in some particular use case?

GRIESKAMP: Right. So that’s why it has

IllustratIon Base D on a Photogra Ph courtesy of BoB BIn Der