(a) Scalpel (b) Bandage Scissors (c) Hook Retractor
(g) Speech On/Off
table 2. Confusion matrix at ς = 0.99 for the gesture lexicon.
97 3 90 6 4 94 3 3 100 100 2 98 5 95 4 96 5 94 1 Scalpel B. Scissors H. Retractor Forceps Hemostat Needle Speech On/Off Scissors Sleep/Wake Retractor
Sx S/W R
table 3. Confusion matrix at γ= 0.5.
100 100 100 77 23 100 100 100 Scalpel Scissors Retractor Hemostat Clipper Forceps Hook
Clip Forceps Hook
rithm we developed. 12 A mask of the
hand is obtained by thresholding the
depth map and the area of the closest blob. The contour and convex hull
of the blob identify the configuration
of the fingers and associate the configuration to a hand in a static pose. 13
A gesture is preceded by the hand in
the static pose, terminating when the
hand is not in the sensor’s field of
view. Hand-posture recognition uses
this movement as a cue to temporally
segment the gesture. The Gestonurse
lexicon for surgical tasks (see Table 1)
is based on standard OR gestures for
requesting surgical instruments. 8, 19
The positions of the localized fingertips are recorded to obtain trajectories
for each fingertip during the gesture.
The trajectories are derived from the
screen coordinates (in pixels) of the
localized fingertip and the depth (in
millimeters) of the fingertip with respect to the depth sensor; see the online Appendix for algorithmic details.
The system recognizes gestures to
within 160ms (real time) from the time
they were performed by the surgeon.
The system’s overall response time
is slightly more than two seconds, including the time needed (160ms) to
recognize the gesture, plus the overhead time the robotic arm needs to
deliver the instruments (two seconds
on average). The time the system needs
includes for the surgeon to physically
perform a gesture, approximately one
to two seconds for experienced users
and five to six seconds for novices.
Instrument localization and recognition. One of the many responsibilities of human surgical scrub techs is
to remove instruments surgeons might
place around the operating area during a procedure; this requires continuously monitoring the area to detect the
presence of instruments. Detection
means the instrument is localized, recognized, and finally removed. The camera monitoring the region is calibrated
with respect to the robot so the coordinates of the instrument in the image
plane can be converted to coordinates
in the robot-frame view.
Robot control scheme. Delivering
and retrieving instruments can be a
risky activity due to potential harm to
a patient and to the surgical team if potential collisions with the surgeon are
not avoided. Imagine the robot passes