retractors and applying sponges to or
suctioning the operative site). A robot
controlled through hand gestures and
speech commands is a natural alternative that does not affect the normal
flow of surgery.
Previous surgical robots include one
used for object retrieval3 and others
with haptic feedback (such as SOFIE24
and the da Vinci surgical system11 for
minimally invasive procedures in endoscopic and laparoscopic surgeries).
Previous robotic scrub nurses in-
clude the voice-controlled “Penelo-
pe,” 15, 23 which localizes, recognizes,
and returns used instruments. Anoth-
er voice-controlled robot, 4 also uses
computer-vision techniques to recog-
nize, deliver, and retrieve surgical in-
struments. A problem with voice-only
systems is performance degradation22
in noisy environments due to, say, the
sound of drills, anesthesia machines,
surgical staff side conversations, and
operating equipment that can compro-
mise patient safety; for example, a sur-
geon might say “ 50,000 units,” but the
anesthetist hears “ 15,000 units.” 1 Er-
rors can have dramatically adverse con-
sequences for a patient’s well-being.
figure 1. Gestonurse robotic assistant.
depth-based action recognition for instrument prediction; its 3D point-esti-mation method requires the surgeon
wear markers of special reflective material for action recognition that could
potentially compromise sterility.
Trauma Pod9 is a mobile-robotic-surgery effort sponsored by the U.S.
Department of Defense intended to
perform life-saving procedures on the
battlefield; it responds only to voice
commands, not physical gesture. Treat
el al. 23 developed a robot that delivers instruments to the main surgeon
following verbal requests, retrieving
them as soon as they are no longer required. The instruments are identified
through machine-vision algorithms,
with decisions made through a cognitive architecture. In 2011, Jacob et
al. 12–14 and Wachs et al. 25 presented Gestonurse, the first surgical scrub nurse
to understand nonverbal communication, including hand gestures.
In this article, we make two main
contributions with respect to previous
work: how verbal and nonverbal information can, when combined, improve
the robustness of a robotic scrub nurse
and how to assess the effectiveness of
the interaction between surgeon and
robot in an OR setting through a mock
surgery—an abdominal incision and
closure using a phantom simulator.
Gestonurse gestures are recognized
from the video/depth stream acquired
by a Microsoft Xbox 360 Kinect sensor; a robotic arm then delivers the re-
figure 2. System overview.
Multimodal Instrument Delivery
Robot Control Scheme
Clustering Convex Hull
Safe Human-Robot Collaboration