signed combinatorial methods that we
call “Segmentation Fusion” [ 3] for the
formation of segments that do not require an initial overpartitioning of the
images. Segmentation Fusion works by
finding a selection of optimal segments
from a set of initial candidate, hypothetical segments that are guaranteed to cover the stack. The selection of segments is
formulated as an integer programming
problem that maximizes the total score
of all the segments for the full stack
while satisfying constraints given by our
prior knowledge about the topology and
geometry of neural networks.
The output of Segmentation Fusion
is a graph of segments for each stack,
where the connected components
correspond to distinct neurons in the
stack. By solving this problem for small
volumes within the original stack and
then matching the results across adjacent volumes, we obtain the high-resolution dense reconstruction of the full
block of tissue. The detailed results obtained with the Segmentation Fusion
method is shown in Figure 6.
THE FUTURE: MINING THE
The connectome project is an ambitious multidisciplinary effort that
aims to apply biology, computer science, and software engineering to the
grand challenge of determining the
detailed neural circuitry of the brain.
Our lab and many others are just starting to address some of the core problems involved in high-throughput and
high-resolution brain reconstruction
and visualization. But this is only the
beginning; the analysis and mining of
the first connectome, namely the first
full connectivity diagram of the brain
of a mammal, is expected to lead to an
unprecedented level of understanding
of the basic cognitive functions and
building blocks of the brain.
We foresee that the datasets result-
ing from the identification of the first
connectomes will open important op-
portunities across different branches
of science. Some hope to learn valuable
information about cognitive develop-
ment and neural disorders; others ex-
pect to learn good design principles for
the synthesis of intelligent machines.
One thing is sure: It will lead to a better
understanding of ourselves.
Figure 6: Results of the fully automatic grouping of pixels into different neurons
using our Segmentation Fusion method for different values of the parameter z.
Amelio Vázquez-Reina is a Ph. D. candidate in computer
science at Tufts University where his advisor is Professor
Eric Miller. Since 2008, Vázquez-Reina has also been a
researcher at the Center for Brain Science at Harvard
University where he works on image segmentation
methods for connectomics under the supervision of
professors Hanspeter Pfister and Jeff Lichtman. His
research interests are in computer vision and machine
Won-Ki Jeong is a research scientist at the Center for
Brain Science at Harvard University. His research interests
include scientific visualization, image processing, and
general purpose computing on the graphics processor in
the field of biomedical image analysis. He received a Ph.D.
Degree in computer science from the University of Utah in
2008, and was a member of the Scientific Computing and
Imaging (SCI) institute at Utah.
Jeff Lichtman M.D., Ph.D. is the Jeremy R. Knowles
Professor of Molecular and Cellular Biology and a member
of the Center for Brain Science at Harvard University. His
interests focus on the development of neural circuits that
are refined by experience during mammalian development.
He has been working on connectomic techniques to render
neural circuits in their entirety with either fluorescence
based (Brainbow) or electron based (AL TUM) tools.
Hanspeter Pfister is the Gordon McKay Professor of the
Practice in the School of Engineering and Applied Sciences
at Harvard University. His research in visual computing lies
at the intersection of visualization, computer graphics,
and computer vision. It spans a wide range of topics,
including bio-medical visualization, 3-D reconstruction,
GPU computing, and data-driven methods in computer
1. Hay worth, K. J., Kasthuri, N., Schalek, R., and
Lichtman, J. W. Automating the collection of ultrathin
serial sections for large volume TEM reconstructions.
Microsc Microanal 12 (2006), 86-7.
2 Jeong, W., Beyer, J., Hadwiger, M., Blue, R., Law, C.,
Vazquez, A., Reid, C., Lichtman, J., and Pfister, H.,
SSECRE T T and Neuro Trace: Interactive visualization
and anaysis tools for large-scale neuroscience
datasets. IEEE Computer Graphics and Applications
30, 3 (2010), 58-70.
3 Jeong, W., Schneider, J., Turney, S. G., Faulkner-Jones,
B.E., Meyer, D., Westermann, R., Reid, R.C., Lichtman,
J.;, and Pfister, H. Interactive Histology of Large-Scale
Biomedical Image Stacks. IEEE Transactions on
Visualization and Computer Graphics 16, 6 (2010),
4 Livet, J., Weissman, T. A., Kang, H., Draft, R. W., Lu,
J., Bennis, R. A., Sanes, J.R., and Lichtman, J. W.
Transgenic strategies for combinatorial expression
of fluorescent proteins in the nervous system. Nature
450, 7166 (2007), 56-62.
5 Lu. J., Tapia, J.C., White, O.L., and Lichtman, J. W. The
interscutularis muscle connectome. PLoS Biology 7, 2
6 Vazquez-Reina, A., Huang, D., Gelbart, M., Miller, E.,
Lichtman, J., and Pfister, H. Segmentation fusion for
connectomics. To appear in ICCV 2011
7 Vazquez-Reina, A., Miller, E., and Pfister, H. Multiphase
geometric couplings for the segmentation of neural
processes. In Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition (2009),
© 2011 ACM 1528-4972/11/0900 $10.00