interface technology are now being
evaluated for integration into C2 systems by the Army’s Program Executive
Office responsible for command-and-control technologies.
We have shown how the STP multimodal interface can address user-interface problems challenging current C2 GUIs. STP is quick and easy to
learn and use and supports many different form factors, including handheld, tablet, vehicle-based, workstation, and ultra-mobile digital paper
and pen. The interface takes advantage of and reinforces skills soldiers
already have, as they are trained in the
standardized language, symbols, and
military decision-making process.
In virtue of this common “doctrinal”
language, STP users can quickly create a course of action or enter data
multimodally for operations, C2, and
simulation systems without extensive
training on a complicated user interface. The result is a highly usable
interface that can be integrated with
existing C2 systems, thus increasing
user efficiency while decreasing cost.
STP development was supported
by Small Business Innovation Research Phase III contracts, including
HR0011-11-C-0152 from DARPA, a
subcontract from SAIC under prime
contract W15P7T-08-C-M011, a subcontract from BAE Systems under
prime contract W15P7T-08-C-M002,
and contract W91CRB-10-C-0210
from the Army Research, Development, and Engineering Command/
Simulation and Training Technology
Center. This article is approved for
public release, distribution unlimited. The results of this research and
the opinions expressed herein are
those of the authors, and not those of
the U.S. Government. We thank Todd
Hughes, Colonels (ret.) Joseph Moore,
Pete Corpac, and James Zanol and the
ROTC student testers. We are grateful to Paulo Barthelmess, Sumithra
Bhakthavatsalam, John Dowding,
Arden Gudger, David McGee, Moiz
Nizamuddin, Michael Robin, Melissa
Trapp-Petty, and Jack Wozniak for
their contributions to developing and
testing of STP. Thanks also to Sharon
Oviatt, General (ret.) Peter Chiarelli,
and the anonymous reviewers.
1. Alberts, D.S. and Hayes, R.E. Understanding Command
and Control. DoD Command and Control Research
Program Publication Series, Washington, D.C., 2006.
2. Bolt, R.A. Voice and gesture at the graphics interface.
ACM Computer Graphics 14, 3 (1980), 262–270.
3. Cheyer, A. and Julia, L. Multimodal maps: An
agent-based approach. In Proceedings of the
International Conference on Cooperative Multimodal
Communication (Eindhoven, the Netherlands, May).
Springer, 1995, 103–113.
4. Clarkson, J.D. and Yi, J. LeatherNet: A synthetic forces
tactical training system for the USMC commander.
In Proceedings of the Sixth Conference on Computer
Generated Forces and Behavioral Representation
Technical Report IS T- TR- 96-18, University of Central
Florida, Institute for Simulation and Training, Orlando,
FL, 1996, 275–281.
5. Cohen, P.R., Johnston, M., McGee, D., Oviatt, S.,
Pittman, J., Smith, I., Chen, L., and Clow, J. QuickSet:
Multimodal interaction for distributed applications.
In Proceedings of the Fifth ACM International
Conference on Multimedia (Seattle, WA, Nov. 9–13).
ACM Press, New York, 1997, 31–40.
6. Cohen, P.R. and McGee, D. R. Tangible multimodal
interfaces for safety-critical applications. Commun.
ACM 47, 1 (Jan. 2004), 41–46.
7. Cohen, P.R., McGee, D., Oviatt, S., Wu, L., Clow, J., King,
R., Julier, S., and Rosenblum, L. Multimodal interaction
for 2D and 3D environments. IEEE Computer Graphics
and Applications 19, 4 (Apr. 1999), 10–13.
8. Courtemanche, A. J. and Ceranowicz, A. ModSAF
development status. In Proceedings of the Fifth
Conference on Computer Generated Forces and
Behavioral Representation, University of Central
Florida, Institute for Simulation and Training, Orlando,
FL, 1995, 3–13.
9. Dowding, J., Gawron, J.M., Appelt, D., Bear, J.,
Cherny, L., Moore, R., and Moran, D. Gemini: A natural
language system for spoken-language understanding.
In Proceedings of the 31st Annual Meeting of the
Association for Computational Linguistics (Ohio State
University, Columbus, OH, June 22–26). Association for
Computational Linguistics, Stroudsburg, PA, 1993, 54–61.
10. Dowding, J., Frank, J., Hockey, B. A., Jonsson, A., Aist,
G., and Hieronymus, J. A spoken-dialogue interface
to the EUROPA planner. In Proceedings of the Third
International NASA Workshop on Planning and
Scheduling for Space ( Washington, D. C.). NASA, 2002.
11. Greene, H., Stotts, L., Patterson, R., and Greenburg, J.
Command Post of the Future: Successful Transition of
a Science and Technology Initiative to a Program of
Record. Defense Acquisition University, Fort Belvoir,
VA, Jan. 2010; http://www.dau.mil
12. Hammond, T., Logsdon, D., Peschel, J., Johnston,
J., Taele, P., Wolin, A., and Paulson, B. A sketch-recognition interface that recognizes hundreds of
shapes in course-of-action diagrams. In Proceedings
of ACM CHI Conference on Human Factors in
Computing Systems (Atlanta, Apr. 10–15). ACM Press,
New York, 2010, 4213–4218.
13. Hyman, P. Speech-to-speech translations stutter, but
researchers see mellifluous future. Commun. ACM 57,
4 (Apr. 2014), 16–19.
14. Johnston, M., Cohen, P. R., McGee, D., Oviatt, S. L.,
Pittman, J.A., and Smith, I. Unification-based multimodal
integration. In Proceedings of the 35th Annual Meeting of
the Association for Computational Linguistics and Eighth
Annual Meeting of the European ACL (Madrid, Spain,
July 7–12). Association for Computational Linguistics,
Stroudsburg, PA, 1997, 281–288.
15. Johnston, M., Bangalore, S., Varireddy, G., Stent, A.,
Ehlen, P., Walker, M., Whittaker, S., and Maloor, P.
MATCH: An architecture for multimodal dialogue
systems. In Proceedings of the 40th Annual Meeting
of the Association for Computational Linguistics
(Philadelphia, PA, July). Association for Computational
Linguistics, Stroudsburg, PA, 2002, 376–383.
16. Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A.,
Li, X., Cohen, P. R., and Feiner, S. Mutual disambiguation
of 3D multimodal interaction in augmented and virtual
reality. In Proceedings of the Seventh International
Conference on Multimodal Interfaces ( Trento, Italy,
Oct. 4–6). ACM Press, New York, 2005, 12–19.
17. Kara, L.B. and Stahovich, T. F. An image-based,
trainable symbol recognizer for hand-drawn sketches.
Computers and Graphics 29, 4 (2005), 501–517.
18. Kumar, S., Cohen, P.R., and Coulston, R. Multimodal
interaction under exerted conditions in a natural field
setting. In Proceedings of the Sixth International
Conference on Multimodal Interfaces (State College,
PA, Oct. 13–15). ACM Press, New York, 2004, 227–234.
19. MacEachren, A.M., Cai, G., Brewer, I., and Chen,
J. Supporting map-based geo-collaboration
through natural interfaces to large-screen display.
Cartographic Perspectives 54 (Spring 2006), 16–34.
20. Moran, D.B., Cheyer, A.J., Julia, L.E., Martin, D.L.,
and Park, S. Multimodal user interfaces in the Open
Agent Architecture. In Proceedings of the Second
International Conference on Intelligent User
Interfaces (Orlando, FL, Jan. 6–9). ACM Press, New
York, 1997, 61–68.
21. Myers, K., Kolojejchick, J., Angiolillo, C., Cummings, T.,
Garvey, T., Gervasio, M., Haines, W., Jones, C., Knittel,
J., Morley, D., Ommert, W., and Potter, S. Learning
by demonstration for military planning and decision
making: A deployment story. In Proceedings of the
23rd Innovative Applications of Artificial Intelligence
Conference (San Francisco, CA, Aug. 6–10). AAAI
Press, Menlo Park, CA, 2011, 1597–1604.
22. O’Hara, K., Gonzalez, G., Sellen, A., Penney, G.,
Varnavas, A., Mentis, H., Criminisi, A., Corish, R.,
Rouncefield, M., Dastur, N., and Carrell, T. Touchless
interaction in surgery. Commun. ACM 57, 1 (Jan.
23. Oviatt, S.L. Multimodal interfaces. The Human-Computer Interaction Handbook: Fundamentals,
Evolving Technologies and Emerging Applications,
Revised Third Edition, J. Jacko, Ed. Lawrence Erlbaum
Associates, Mahwah, NJ, 2012, 405–430.
24. Oviatt, S.L. Taming recognition errors with a
multimodal architecture. Commun. ACM 43, 9 (Sept.
25. Oviatt, S.L. and Cohen, P.R. Perceptual user interfaces:
Multimodal interfaces that process what comes
naturally. Commun. ACM 43, 3 (Mar. 2000), 45–53.
26. Oviatt, S.L. and Cohen, P.R. The Paradigm Shift to
Multimodality in Contemporary Computer Interfaces.
Morgan & Claypool Publishers, San Francisco, CA, 2015.
27. Stilman, B., Yakhnis, V., and Umanskiy, O. Strategies
in large-scale problems. In Adversarial Reasoning:
Computational Approaches to Reading the Opponent’s
Mind, A. Kott and W. McEneaney, Eds. Chapman &
Hall/CRC, London, U. K., 2007, 251–285.
28. U. S. Army. U. S. Army Field Manual 101–5-1, Chapter
5, 1997; http://armypubs.army.mil/doctrine/dr_pubs/
Philip R. Cohen ( firstname.lastname@example.org) is a co-founder
of Adapx, a fellow of the Association for the Advancement
of Artificial Intelligence, and a past-president of the
Association for Computational Linguistics.
Edward C. Kaiser ( email@example.com) is a senior
application engineer at Sensory Inc., Portland, OR, and
was co-PI/PI (2008–2009/2010) for STP at Adapx Inc.,
M. Cecelia Buchanan ( firstname.lastname@example.org) is a
consultant at Tuatara Consulting, Seattle, WA, and was a
research scientist at Adapx, Seattle, WA, when this article
Scott Lind ( email@example.com) is vice president for
Department of Defense and federal solutions at Adapx,
Michael J. Corrigan ( firstname.lastname@example.org) is a
research software engineer at Adapx, Seattle, WA.
R. Matthews Wesson ( email@example.com) is a
senior research programmer at Adapx, Seattle, WA.
2015 ACM 0001-0782/15/04 $15.00
Watch the authors discuss
this work in this exclusive