training for each and every new person
our vendors in China and India hired to
perform the testing for us. That training covered not only our model-based
testing approach, but also some other
aspects of the overall methodology.
BInDER: How long did it take for
moderately competent developers who
had never encountered model-based
testing before to get to the point where
they could actually be pretty productive?
KICILLoF: On average, I’d say that
took close to a month.
BInDER: Once your testers were
trained, how did your testing approach
evolve? Did you run into any significant
problems along the way?
GRIESKAMP: It proved to be a fairly
smooth transition since we were just
working with concepts that were part
of the prototype we had already developed back at Microsoft Research.
That said, it actually was just a prototype when this team took it over, so
our main challenge was to stabilize the
technology. You know how prototypes
are—they crash and you end up having
to do workarounds and so forth. We’ve
had a development team working to
improve the tool over the past three
years, and thousands of fixes have
come out of that.
Another potential issue had to do
with something that often crops up in
model-based testing: a state-explosion
problem. Whenever you model—if
you naively define some rules to update your state whenever certain con-
how did you
manage to convince
could take several
had no experience
in this area and
a fairly esoteric
into rule systems?
ditions are met and then you just let
the thing run—there’s a good chance
you’re going to end up getting overrun
by all those state updates. For example,
when using this tool, if you call for an
exploration, that should result in a visualization of the exploration graph
that you can then inspect. If you’re
not careful, however, you could end
up with thousands and thousands of
states the system will try to explore for
you. There’s just no way you’re going to
be able to visualize all of that.
Also, in order to see what’s actually
going on, you need to have some way of
pruning down the potential state space
such that you can slice out those areas
you know you’re going to need to test.
That’s where one of our biggest challenges was: finding the right way to
slice the model.
The idea here was to find the right
slicing approach for any given problem, and the tool provides a lot of assistance for accomplishing that. It didn’t
come as a surprise to us that this issue
of finding the right way to slice the
space would end up being a problem—
we had expected that. We actually had
already added some things to the tool
to deal with that, which is probably one
of the reasons the project has proved to
be a success.
KICILLoF: The secret is to use test purposes as the criterion for slicing.
BInDER: With that being only a subset of all the behaviors you would be
looking at in some particular use case?
GRIESKAMP: Right. So that’s why it has
IllustratIon Base D on a Photogra Ph courtesy of BoB BIn Der