With the GPU version of cv::resize
we were able to decrease scaling time
from 41 milliseconds to 26 milliseconds for each input frame, which is
equal to 1. 6 times local speedup. Because of the GPU implementation of
image warping, we could achieve even
better local improvements—a boost of
8–14 times in performance, depending
on the projection type. As a result, total
application speedup was 1. 5–2.0 times,
meeting performance requirements.
Video stabilization. One of the negative consequences of recording video
without a tripod is camera shake,
which significantly degrades the
viewing experience. To achieve visually pleasant results, all movements
should be smooth, and the high-frequency variations in camera orientation and translation must be filtered.
Numerous approaches have been
developed, some have become open
source or commercially available
tools. There exist computationally intensive approaches offline that take a
considerable amount of time, while
the lightweight online algorithms
are more suitable for mobile devices.
High-end approaches often reconstruct the 3D movement of the camera and apply sophisticated nonrigid
image warping to stabilize the video.
On mobile devices more lightweight
approaches using translation, affine
warping, or planar perspective transformations may make more sense.
We experimented with translation
and affine models, and in both cases
the GPU was able to eliminate the
major hotspot, which was the application of the compensating transformation to an input frame. Applying
translation to compensate for the motion simply means shifting the input
frame along the x and y axes and cutting off some of the boundary areas
for which some of the frames now do
not contain color information (see
In terms of programming, one
should choose a properly located submatrix and then resize it into a new
image at the same resolution as the
original video stream, as suggested
in Figure 11. Surprisingly, this simple
step consumed more than 140 milliseconds. Our GPU GLSL implementation was five to six times faster than
C++ and took about 25 milliseconds.
Nevertheless, 25 milliseconds is
still too long for a real-time algorithm,
which is why we next tried to obtain
more speed from asynchronous calls.
A special class was created for stabilizing frames on the GPU. This class
immediately returns a result from the
previous iteration stored in its image-buffer field and creates a TBB::task
for processing the next frame. As a
result, GPU processing is performed
in the background, and the apparent
cost and delay for the caller is equal
to just copying a full frame. This trick
was also applied to an expensive col-or-conversion procedure, and with
further optimizations of the memory-access patterns, we achieved real-time
GPUs were originally developed to accelerate the conversion of 3D scene
descriptions into 2D images at interactive rates, but as they have become
more programmable and flexible, they
have also been used for the inverse task
of processing and analyzing 2D images
and image streams to create a 3D description, to control some application
so it can react to the user or events in
the environment, or simply to create
higher-quality images or videos. As
computer-vision applications become
more commonplace, it will be interesting to see whether a different type of
computer-vision processor that would
be even more suitable for image processing is created to work with a GPU,
or whether the GPU remains suitable
even for this task. The current mobile
GPUs are not yet as flexible as those on
larger computers, but this will change
OpenCV (and other related APIs
such as Point Cloud Library) have
made it easier for application develop-
ers to use computer vision. They are
well-documented and vibrant open
source projects that keep growing,
and they are being adapted to new
computing technologies. Examples of
this evolution are the transition from
a C to a C++ API in OpenCV and the ap-
pearance of the OpenCV GPU module.
The basic OpenCV architecture, how-
ever, was designed mostly with CPUs
in mind. Maybe it is time to design a
new API that explicitly takes heteroge-
neous multiprocessing into account,
where the main program may run on
a CPU or several CPUs, while major
parts of the vision API run on differ-
ent types of hardware: a GPU, a DSP
(digital signal processor), or even a
dedicated vision processor. In fact,
Khronos has recently started working
on such an API, which could work as
an abstraction layer that allows inno-
vation independently on the hardware
side and allows for high-level APIs
such as OpenCV to be developed on
top of this layer while being somewhat
insulated from the changes in the un-
derlying hardware architecture.
We thank Colin Tracey and Marina
Kolpakova for help with power analysis; Andrey Pavlenko and Andrey Ka-maev for GLSL and NEON code; and
Shalini Gupta, Shervin Emami, and Michael Stewart for additional comments.
NVIDIA provided support, including
hardware used in the experiments.
1. khronos openMaX standard; http://www.khronos.org/
2. liu, F., Gleicher, M., Wang, J., Jin, H., agar wala, a.
subspace video stabilization. ACM Transactions on
Graphics 30, 1 (2011), 4:1–4: 10.
3. Matsushita, y., ofek, e., Ge, W., tang, X., shum, H.-y.
Full-frame video stabiliza¬tion with motion inpainting.
IEEE Transactions on Pattern Analysis and Machine
Intelligence 28, 7 (2006), 1150–1163.
4. newcombe, r.a., Izadi, s. et al. kinectfusion: real-time dense surface mapping and tracking. IEEE
International Symposium on Mixed and Augmented
Reality (2011), 127–136.
5. openCV library; http://code.opencv.org.
6. Point Cloud library; http://pointclouds.org.
7. szeliski, r. Image alignment and stitching: a tutorial.
Foundations and Trends in Computer Graphics and
Vision 2, 1 (2006), 1–104.
8. Willow Garage. robot operating system; http://www.
Kari Pulli is a senior director at nVIdIa research, where
he heads the Mobile Visual Computing research team and
works on topics related to cameras, imaging, and vision
on mobile devices. He has worked on standardizing mobile
media aPIs at khronos and JCP and was technical lead of
the digital Michelangelo Project at stanford university.
Anatoly Baksheev is a project manager at Itseez. He
started his career there in 2006 and was the principal
developer of multi-projector system argus Planetarium.
since 2010 he has been the leader of the openCV GPu
project. since 2011 he has work on the GPu acceleration
module for Point Cloud library.
Kirill Kornyakov is a project manager at Itseez, where
he leads the development of openCV library for mobile
devices. He manages activities on mobile operating-system support and computer-vision applications
development, including performance optimization for
nVIdIa tegra platform.
Victor Eruhimov is Cto of Itseez. Prior to co-founding
the company, he worked as a project manager and senior
research scientist at Intel, where he applied computer-vision and machine-learning methods to automate Intel
fabs and revolutionize data processing in semiconductor
© 2012 aCM 0001-0782/12/06 $10.00