How does the model29 perform real-world recognition tasks? And how
does it compare to state-of-the-art
artificial-intelligence systems? Given
the specific biological constraints the
theory must satisfy (such as using only
biophysically plausible operations,
receptive field sizes, and a range of invariances), it was not clear how well the
model implementation would perform
compared to systems heuristically engineered for these complex tasks.
Several years ago, we were surprised
to find the model capable of recognizing complex images, 27 performing at a
level comparable to some of the best
existing systems on the CalTech- 101
image database of 101 object categories with a recognition rate of about
55% (chance level < 1%); see Serre et
al. 27 and Mutch and Lowe. 19 A related
system with fewer layers, less invariance, and more units had an even better recognition rate on the CalTech
data set. 20
We also developed an automated
system for parsing street-scene images27 based in part on the class of
models described earlier. The system
recognizes seven different object categories—cars, pedestrians, bikes,
skies, roads, buildings, trees—from
natural images of street scenes despite very large variations in shape
(such as trees in summer and winter
and SUVs and compact cars from any
point of view).
Content-based recognition and
search in videos is an emerging application of computer vision, whereby
neuroscience may again suggest an
avenue for approaching the problem.
In 2007, we developed an initial model for recognizing biological motion
and actions from video sequences
based on the organization of the dorsal stream of the visual cortex, 13 which
is critically linked to the processing
of motion information, from V1 and
MT to higher motion-selective areas
MST/FST and STS. The system relies
on computational principles similar
to those in the model of the ventral
stream described earlier but that start
with spatio-temporal filters modeled
after motion-sensitive cells in the primary visual cortex.
We evaluated system performance
for recognizing actions (human and
animal) in real-world video sequenc-
Black corresponds to data used to derive the parameters of the model, red to data
consistent with the model (not used to fit model parameters), and blue to actual
correct predictions by the model. notations: PFc (prefrontal cortex), v1 (visual
area i or primary visual cortex), v4 (visual area iv), and it (inferotemporal cortex).
Data from these areas corresponds to monkey electrophysiology studies. lOc (lateral
Occipital complex) involves fMri with humans. the psychological studies are
psychophysics on human subjects.
area type of data
Psych. rapid animal categorization
Face inversion effect
Loc Face processing (fMrI)
Pfc differential role of IT and PFc in categorization
it Tuning and invariance properties
read out for object category
Average effect in IT
V4 MAX operation
Tuning for two-bar stimuli
Tuning for boundary conformation
Tuning for cartesian and non-cartesian gratings
V1 Simple and complex cells tuning properties
MAX operation in subset of complex cells
Ref. biol. data Ref. model data
( 1) ( 1)
( 2) ( 2)
( 3) ( 3)
( 4) ( 5)
( 6) ( 5)
( 7) ( 8, 9)
( 10) ( 10)
( 11) ( 5)
( 12) ( 8, 9)
( 13) ( 8)
( 14) ( 8, 15)
( 16) ( 8)
( 17–19) ( 8)
( 20) ( 5)
1. serre, t., oliva, a., and Poggio, t. Proc. Natl. Acad. Sci. 104, 6424 (apr. 2007).
2. riesenhuber, M. et al. Proc. Biol. Sci. 271, s448 (2004).
3. Jiang, X. et al. Neuron 50, 159 (2006).
4. Freedman, d.J., riesenhuber, M., Poggio, t., and Miller, e. K. Journ. Neurosci. 23, 5235 (2003).
5. riesenhuber, M. and Poggio, t. Nature Neuroscience 2, 1019 (1999).
6. Logothetis, n.K., Pauls, J., and Poggio, t. Curr. Biol. 5, 552 (May 1995).
7. hung, c. P., Kreiman, g., Poggio, t., and dicarlo, J. J. Science 310, 863 (nov. 2005).
8. serre, t. et al. MI T AI Memo 2005-036 / CBCL Memo 259 (2005).
9. serre, t. et al. Prog. Brain Res. 165, 33 (2007).
10. Zoccolan, d., Kouh, M., Poggio, t., and dicarlo, J. J. Journ. Neurosci. 27, 12292 (2007).
11. gawne, t. J. and Martin, J.M. Journ. Neurophysiol. 88, 1128 (2002).
12. reynolds, J.h., chelazzi, L., and desimone, r. Journ. Neurosci. 19, 1736 (Mar. 1999).
13. taylor, K., Mandon, s., Freiwald, w.a., and Kreiter, a.K. Cereb. Cortex 15, 1424 (2005).
14. Pasupathy, a. and connor, c. Journ. Neurophysiol. 82, 2490 (1999).
15. cadieu, c. et al. Journ. Neurophysiol. 98, 1733 (2007).
16. gallant, J. L. et al. Journ. Neurophysiol. 76, 2718 (1996).
17. schiller, P.h., Finlay, b. L., and Volman, s. F. Journ. Neurophysiol. 39, 1288 (1976).
18. hubel, d.h. and wiesel, t.n. Journ. Physiol. 160, 106 (1962).
19. de Valois, r. L., albrecht, d.g., and thorell, L.g. Vision Res. 22, 545 (1982).
20. Lampl, i., Ferster, d., Poggio, t., and riesenhuber, M. Journ. Neurophysiol. 92, 2704 (2004).
es, 13 finding that the model of the dor-
sal stream competed with a state-of-
the-art action-recognition system (that
outperformed many other systems) on
all three data sets. 13 A direct extension
of this approach led to a computer sys-
tem for the automated monitoring and
analysis of rodent behavior for behav-
ioral phenotyping applications that
perform on par with human manual
scoring. We also found the learning in
this model produced a large dictionary
of optic-flow patterns that seems con-
sistent with the response properties of
cells in the medial temporal (MT) area
in response to both isolated gratings
and plaids, or two gratings superim-
posed on one another.
Demonstrating that a model designed
to mimic known anatomy and physiol-