Society | DOI: 10.1145/1610252.1610261
Samuel Greengard

making automation Work

Today’s automated systems provide enormous safety and convenience.

However, when glitches, problems, or breakdowns occur, the results can be catastrophic.

It’S no SeCret that engineers and designers constantly seek to build safer and more con- venient systems. And, over the last century, planes, trains, automobiles, and industrial machines have become far more automated and efficient. However, when a Metro subway train rammed into another train in Washington, D.C. last June, designers had to confront the unpleasant reality that automation may have been the cause. The accident, which killed nine people and injured 80, may have been rooted in a computer malfunction and the operator’s inability to manually apply the brakes quickly enough.

The Metro train accident lies at the heart of what human factors experts refer to as the “automation paradox.” As automated systems become increasingly reliable and efficient, the more likely it is that human operators will mentally “switch off” and rely upon the automated system. And as the automated system becomes more complex, the odds of an accident or mishap may diminish, but the severity of a failure is often amplified.

As John D. Lee, a professor of industrial and systems engineering at the University of Wisconsin at Madison told the Washington Post: “The better you make the automation, the more difficult it is to guard against these catastrophic failures....”

Understanding how people and machines interact is infinitely complex. Programming all the various possibilities and scenarios into a system can tax even the best design and engineering experts. What’s more, as technology evolves, the entire process grows more convoluted and iterative. In some cases, experts say, it’s wise to ask what purpose automation serves and when it’s best to use it and eschew it.

What is the fallout from automation glitches? Where do programmers, designers, and engineers typically fall short? And what can technologists do

the June 22, 2009 metro subway train crash in Washington, D.C., in which nine people died and 80 were injured, is the deadliest accident in the metro’s 33-year history.

 

to build better systems? There are no simple solutions. But as Donald Norman, professor of computer science at Northwestern University, co-founder of Neilson Norman Group, and author of The Design of Future Things, says, “ Designers often make assumptions or act on incomplete information. They simply don’t anticipate how systems will be used and how unanticipated events and consequences will occur.”

human-machine interface

It’s clear that automation has provided enormous gains to society. Safer and more efficient factories; faster police, emergency, and fire response; and more user-friendly and safer automobiles are only a few of the benefits.

Yet, at the same time, it takes little effort to find evidence of breakdowns between human and machine.

The crash of Air France Flight 447 that occurred over the Atlantic Ocean last June—killing all 228 people aboard—may have been caused by a malfunction in a speed sensor. The plane’s Pitot tubes, a pressure measurement instrument used to track

fluid flow velocity, may have became blocked by ice. At that point, they may have stopped emitting signals, and experts say that the pilots could have encountered false speed readings. In fact, the jet—which was coping with a series of storms, including a severe thunderstorm—reportedly relayed a signal that its computer system no longer knew the speed of the aircraft, and that automatic pilot and thrust functions were switched off. This may have forced the pilots to take over manual control during chaotic, if not impossible, flying conditions.

There are also plenty of examples of humans having trouble with automation systems in everyday life. As automobiles become more automated, new problems crop up. For instance, motorists blindly follow the incorrect directions provided by a navigation system, even though a glance at the road would indicate there’s an obvious error. A few motorists have even driven off a cliff or into oncoming traffic after following directions explicitly. What’s more, studies show that many motorists use automation features, such as