Gestonurse is a multimodal robotic
scrub nurse we developed at Purdue
and the Indiana University School of
Medicine to reliably pass surgical instruments to surgeons and other members of a human surgical team, yielding
gesture-recognition accuracy of 95.96%
on average. The related Kinect-based
robotic vision system we developed
recognizes and picks instruments with
a recognition rate of 92.38%. We also
developed an instrument-picking system
with 100% accuracy and a related disposal system with 92.83% accuracy
(on average), as well as a field-path-planning algorithm to maximize
safety in human-robot collaboration,
implementing it in Gestonurse.
We conducted experiments on the
effects of modality training for participants in a mock surgical procedure,
calculating pre-task training delivered
12.92% reduced task-completion time
on average across both speech and
gesture. We also showed the system
(following user training in both speech
and gesture) is 14.9% faster than
speech only, on average.
Future work on Gestonurse aims to
fuse speech- and gesture-recognition
data in a probabilistic fashion and
transition to a real surgical setting
involving animals at the Veterinary
School at Purdue University in West
Lafayette, IN. Another goal is to add
the ability to predict the next surgical instrument likely to be needed according to the type of procedure (the
context) instead of on a subjective,
variable chain of verbal commands.
We also aim to improve specific features of the system; for example, we
assumed instruments were placed
by surgical staff in fixed positions in
a Mayo tray, with instrument coordinates saved as trajectory points in the
teach pendant’s (the robot’s remote
control) memory. Future versions
will use the algorithm we developed
to automatically detect and pick instruments, regardless of location.
This will require minimal work since
the current design already uses these
techniques to retrieve instruments.
This project was funded, in part, by
the Indiana Clinical and Translation-
al Sciences Institute, by Grant Num-
ber Grant #TR000006 from the Na-
tional Institutes of Health, National
Center for Advancing Translational
Sciences, Clinical and Translational
Sciences Award. We also thank Dr.
Steve Adams of Purdue University for
his consultation and use of the Veteri-
nary School OR, Dr. Rebecca Packer
of Purdue for the surgical supplies we
used in the experiment, and Hairong
Jiang of Purdue for her support im-
plementing the Gestonurse obstacle-
1. beyea, s.c. noise: a distraction, interruption, and
safety hazard. AORN Journal 86, 2 (2007), 281–285.
2. bolt, r.a. Put-that-there: voice and gesture at the
graphics interface. Commun. ACM 14, 3 (1980),
3. borenstein, j. and koren, y. a mobile platform for
nursing robots. IEEE Transactions on Industrial
Electronics 2 (2007), 158–165.
4. carpintero, e., Perez, c., morales, r., garcia, n.,
candela, a., and azorin, j. Development of a robotic
scrub nurse for the operating theatre. In Proceedings
of the Third IEEE International Conference on
Biomedical Robotics and Biomechatronics (elche,
spain, sept. 26–29, 2010), 504–509.
5. carthey j., de laval, m.r., Wright, D.j. et al. behavioral
markers of surgical excellence. Safety Science 41, 5
6. egorova, n.n., moskowitz, a., gelijns, a. et al. managing
the prevention of retained surgical instruments: What
is the value of counting? Annals of Surgery 247, 1
7. firth-cozens, j. Why communication fails in the
operating room. Quality Safety Health Care 13, 5 (oct.
8. fulchiero, g.j., vujevich, j.j., and goldberg, l.h.
nonverbal hand signals: a tool for increasing patient
comfort during dermatologic surgery. Dermatological
Surgery 35, 5 (2009), 856–857.
9. garcia, P., rosen, j., kapoor, c., noakes, m., elbert, g.,
treat, m., ganous, t., hanson, m., manak, j., hasser,
c., rohler, D., and satava, r. trauma pod: a semi-automated telerobotic surgical system. International
Journal of Medical Robotics and Computer-Assisted
Surgery 5, 2 (2009), 136–146.
10. halverson, a.l., casey, j.t., andersson, j., anderson,
k., Park, c., rademaker, a. W., and moorman, D.
communication failure in the operating room. Surgery
149, 3 (2010), 305–310.
11. Intuitive surgical. da vinci surgical system; http://
12. jacob, m.g., li, y., akingba, g., and Wachs, j.P.
gestonurse: a robotic surgical nurse for handling
surgical instruments in the operating room. Journal of
Robotic Surgery 6, 1, (mar. 2012), 53–63.
13. jacob, m.g., li, y., and Wachs, j.P. a gesture-driven
robotic scrub nurse. In Proceedings of the IEEE
International Conference on Systems, Man, and
Cybernetics (anchorage, ak, oct. 9–12, 2011),
14. jacob, m, li, y.t., and Wachs, j.P. gestonurse: a
multimodal robotic scrub nurse. In Proceedings of
the Seventh ACM/IEEE International Conference on
Human Robot Interaction (boston, ma, mar. 5–8).
acm Press, new york, 2012, 153–154.
15. kochan, a. scalpel please, robot: Penelope’s debut in
the operating room. Industrial Robot: An International
Journal 32, 6 (2005), 449–451.
16. kohn, l.t., corrigan, j., and Donaldson, m.s. To Err Is
Human: Building a Safer Health System, Volume 6.
joseph henry Press, 2000.
17. li, y.t., jacob, m., akingba, g., and Wachs, j. P. a
cyber-physical management system for delivering and
monitoring surgical instruments in the or. Surgical
Innovation (Pubmed: 23037804); http://sri.sagepub.
18. lingard l., espin s., Whyte s. et al. communication
failures in the operating room: an observational
classification of recurrent types and effects. Quality
Safety Health Care 13, 5 (2004), 330–334.
Mithun George Jacob ( email@example.com) is a
Ph. D. student in the Department of Industrial engineering
at Purdue university, West lafayette, In.
Yu-Ting Li ( firstname.lastname@example.org) is a Ph.D. student in
the school of Industrial engineering at Purdue university,
West lafayette, In.
George A. Akingba ( email@example.com) is an assistant
professor of surgery and biomedical engineering in the
Division of vascular surgery at the Indiana university
school of medicine, Indianapolis, In.
Juan P. Wachs ( firstname.lastname@example.org) is an assistant
professor in the Industrial engineering school at Purdue
university, director of the Intelligent systems and
assistive technologies lab at Purdue, and affiliated with
the regenstrief center for healthcare engineering, West