works (CNNs) suddenly made it possible to explore scenes and scenarios in
deeper and broader ways. By tossing
vast numbers of images at the artificial
neural network—stop signs, traffic signals, road markings, barriers, trees,
dogs, pedestrians, other vehicles, and
much more—and comparing actions
and reactions such as steering, braking, and acceleration, it’s possible to
cycle rapidly through an array of events
and scenarios en route to more refined
algorithms and better performing self-driving cars.
Of course, the allure of this approach is that in the virtual world, cars
never run out of fuel or need new tires,
and they’re able to log millions of miles
in a single day. There are no fatigued
drivers and no risk of real-world collisions or injuries. However, the benefits
don’t stop there.
“One can say that the real world is
richer in terms of character than the
virtual world, but in the virtual world
All of this is leading researchers
down a different path: the use of game
simulations and machine learning to
build better algorithms and smarter ve-
hicles. By compressing months or
years of driving into minutes or even
seconds, it is possible to learn how to
better react to the unknown, the unex-
pected, and unforeseen, whether it is a
stop sign obscured by graffiti, a worn or
missing lane marking, or snow cover-
ing the road and obscuring everything.
“A human could analyze a situation
and adapt quickly. But an autonomous
vehicle that doesn’t detect something
correctly could produce a result ranging
from annoying to catastrophic,” explains Julian Togelius, associate professor of computer science and engineering at New York University (NYU).
The use of computer games and simulations—including the likes of open-source TORCS (The Open Racing Car
Simulator) and commercially available
Grand Theft Auto V—already is revolutionizing the way researchers develop
autonomous vehicles, as well as robots,
drones, and other machine systems.
Not only is it possible to better understand machine behavior—including
how sensors view and read the surrounding environment—it offers insights into human behavior in different
situations. “These games offer extremely rich environments that allow
you to drive through a broad range of
road conditions that would be difficult
to duplicate in the physical world,” says
Artur Filipowicz, a recent graduate in
operations research and financial engineering at Princeton University who
has used machine learning to advance
research on autonomous vehicles.
The Road Less Traveled
Although the idea of using video game
simulations and AI to boost real-world
performance for autonomous vehicles
has been around for more than a decade, the concept has zoomed forward
over the last few years. The rise of
graphics processing units (GPUs) and
the advent of convolutional neural net-
Technology | DOI: 10.1145/3148817 Samuel Greengard
Game simulations are driving improvements in
machine learning for autonomous vehicles and other devices.
A scene from Rockstar Games’ Grand Theft Auto V, which is helping to revolutionize how
researchers develop autonomous vehicles.