Conclusion

Gestonurse is a multimodal robotic scrub nurse we developed at Purdue and the Indiana University School of Medicine to reliably pass surgical instruments to surgeons and other members of a human surgical team, yielding gesture-recognition accuracy of 95.96% on average. The related Kinect-based robotic vision system we developed recognizes and picks instruments with a recognition rate of 92.38%. We also developed an instrument-picking system with 100% accuracy and a related disposal system with 92.83% accuracy (on average), as well as a field-path-planning algorithm to maximize safety in human-robot collaboration, implementing it in Gestonurse.

We conducted experiments on the effects of modality training for participants in a mock surgical procedure, calculating pre-task training delivered 12.92% reduced task-completion time on average across both speech and gesture. We also showed the system (following user training in both speech and gesture) is 14.9% faster than speech only, on average.

Future work on Gestonurse aims to fuse speech- and gesture-recognition data in a probabilistic fashion and transition to a real surgical setting involving animals at the Veterinary School at Purdue University in West Lafayette, IN. Another goal is to add the ability to predict the next surgical instrument likely to be needed according to the type of procedure (the context) instead of on a subjective, variable chain of verbal commands. We also aim to improve specific features of the system; for example, we assumed instruments were placed by surgical staff in fixed positions in a Mayo tray, with instrument coordinates saved as trajectory points in the teach pendant’s (the robot’s remote control) memory. Future versions will use the algorithm we developed to automatically detect and pick instruments, regardless of location. This will require minimal work since the current design already uses these techniques to retrieve instruments.

acknowledgments

This project was funded, in part, by
the Indiana Clinical and Translation-
al Sciences Institute, by Grant Num-
ber Grant #TR000006 from the Na-
tional Institutes of Health, National
Center for Advancing Translational
Sciences, Clinical and Translational
Sciences Award. We also thank Dr.
Steve Adams of Purdue University for
his consultation and use of the Veteri-
nary School OR, Dr. Rebecca Packer
of Purdue for the surgical supplies we
used in the experiment, and Hairong
Jiang of Purdue for her support im-
plementing the Gestonurse obstacle-
avoidance module.

References

1. beyea, s.c. noise: a distraction, interruption, and safety hazard. AORN Journal 86, 2 (2007), 281–285.

2. bolt, r.a. Put-that-there: voice and gesture at the graphics interface. Commun. ACM 14, 3 (1980), 262–270.

3. borenstein, j. and koren, y. a mobile platform for nursing robots. IEEE Transactions on Industrial Electronics 2 (2007), 158–165.

4. carpintero, e., Perez, c., morales, r., garcia, n., candela, a., and azorin, j. Development of a robotic scrub nurse for the operating theatre. In Proceedings of the Third IEEE International Conference on Biomedical Robotics and Biomechatronics (elche, spain, sept. 26–29, 2010), 504–509.

5. carthey j., de laval, m.r., Wright, D.j. et al. behavioral markers of surgical excellence. Safety Science 41, 5 (2003), 409–425.

6. egorova, n.n., moskowitz, a., gelijns, a. et al. managing the prevention of retained surgical instruments: What is the value of counting? Annals of Surgery 247, 1 (2008), 13-18.

7. firth-cozens, j. Why communication fails in the operating room. Quality Safety Health Care 13, 5 (oct. 2004), 327.

8. fulchiero, g.j., vujevich, j.j., and goldberg, l.h. nonverbal hand signals: a tool for increasing patient comfort during dermatologic surgery. Dermatological Surgery 35, 5 (2009), 856–857.

9. garcia, P., rosen, j., kapoor, c., noakes, m., elbert, g., treat, m., ganous, t., hanson, m., manak, j., hasser, c., rohler, D., and satava, r. trauma pod: a semi-automated telerobotic surgical system. International Journal of Medical Robotics and Computer-Assisted Surgery 5, 2 (2009), 136–146.

10. halverson, a.l., casey, j.t., andersson, j., anderson, k., Park, c., rademaker, a. W., and moorman, D. communication failure in the operating room. Surgery 149, 3 (2010), 305–310.

11. Intuitive surgical. da vinci surgical system; http:// www.intuitivesurgical.com/

12. jacob, m.g., li, y., akingba, g., and Wachs, j.P. gestonurse: a robotic surgical nurse for handling surgical instruments in the operating room. Journal of Robotic Surgery 6, 1, (mar. 2012), 53–63.

13. jacob, m.g., li, y., and Wachs, j.P. a gesture-driven robotic scrub nurse. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (anchorage, ak, oct. 9–12, 2011), 2039–2044.

14. jacob, m, li, y.t., and Wachs, j.P. gestonurse: a multimodal robotic scrub nurse. In Proceedings of the Seventh ACM/IEEE International Conference on Human Robot Interaction (boston, ma, mar. 5–8). acm Press, new york, 2012, 153–154.

15. kochan, a. scalpel please, robot: Penelope’s debut in the operating room. Industrial Robot: An International Journal 32, 6 (2005), 449–451.

16. kohn, l.t., corrigan, j., and Donaldson, m.s. To Err Is Human: Building a Safer Health System, Volume 6. joseph henry Press, 2000.

17. li, y.t., jacob, m., akingba, g., and Wachs, j. P. a cyber-physical management system for delivering and monitoring surgical instruments in the or. Surgical Innovation (Pubmed: 23037804); http://sri.sagepub. com/content/early/2012/10/03/1553350612459109. abstract?rss= 1

18. lingard l., espin s., Whyte s. et al. communication
failures in the operating room: an observational
classification of recurrent types and effects. Quality
Safety Health Care 13, 5 (2004), 330–334.

 

Mithun George Jacob ( [e-mail]) is a Ph. D. student in the Department of Industrial engineering at Purdue university, West lafayette, In. Yu-Ting Li ( [e-mail]) is a Ph.D. student in the school of Industrial engineering at Purdue university, West lafayette, In. George A. Akingba ( [e-mail]) is an assistant professor of surgery and biomedical engineering in the Division of vascular surgery at the Indiana university school of medicine, Indianapolis, In. Juan P. Wachs ( [e-mail]) is an assistant professor in the Industrial engineering school at Purdue university, director of the Intelligent systems and assistive technologies lab at Purdue, and affiliated with the regenstrief center for healthcare engineering, West lafayette, In.