figure 6. Randomized Decision forests. a forest is an ensemble of
trees. each tree consists of split nodes (blue) and leaf nodes (green).
the red arrows indicate the different paths that might be taken by
different trees for a particular input.
. . .
Mean shift is used to find modes in this density efficiently.
All pixels above a learned probability threshold lc are used
as starting points for part c. A final confidence estimate is
given as a sum of the pixel weights reaching each mode. This
proved more reliable than taking the modal density
The detected modes lie on the surface of the body. Each
mode is therefore pushed back into the scene by a learned z
offset zc to produce a final joint position proposal. This simple, efficient approach works well in practice. The bandwidths bc, probability threshold lc, and surface-to-interior
z offset zc are optimized per-part on a hold-out validation set
of 5000 images by grid search. (As an indication, this resulted
in mean bandwidth 0.065m, probability threshold 0.14, and
z offset 0.039m).
training. Each tree is trained on a different set of randomly
synthesized images. A random subset of 2000 example pixels from each image is chosen to ensure a roughly even distribution across body parts. Each tree is trained using the
algorithm in Lepetit et al. 11 To keep the training times down,
we employ a distributed implementation. Training three
trees to depth 20 from 1 million images takes about a day on
a 1000 core cluster.
In this section, we describe the experiments performed to
evaluate our method. We show both qualitative and quanti-
tative results on several challenging datasets and compare
with both nearest-neighbor approaches and the state of the
art. 8 We provide further results in the supplementary mate-
rial. Unless otherwise specified, parameters below were set
as 3 trees, 20 deep, 300 k training images per tree, 2000 train-
ing example pixels per image, 2000 candidate features f, and
50 candidate thresholds t per feature.
test data. We use challenging synthetic and real depth
images to evaluate our approach. For our synthetic test set,
we synthesize 5000 depth images, together with the ground
truth body parts labels and joint positions. The original
mocap poses used to generate these images are held out
from the training data. Our real test set consists of 8808
frames of real depth images over 15 different subjects, hand-
labeled with dense body parts and seven upper body joint
positions. We also evaluate on the real depth data from
Ganapathi et al. 8 The results suggest that effects seen on syn-
thetic data are mirrored in the real data, and further that our
synthetic test set is by far the ‘hardest’ due to the extreme
variability in pose and body shape. For most experiments,
we limit the rotation of the user to ± 120° in both training
and synthetic test data, since the user faces the camera (0°)
in our main entertainment scenario, though we also evalu-
ate the full 360° scenario.
error metrics. We quantify both classification and
joint prediction accuracy. For classification, we report the
average per-class accuracy: the average of the diagonal of
the confusion matrix between the ground truth part label
and the most likely inferred part label. This metric
weights each body part equally despite their varying sizes,
though mislabelings on the part boundaries reduce the
For joint proposals, we generate recall-precision curves
as a function of confidence threshold. We quantify accuracy
as average precision per joint, or mean average precision
(mAP) over all joints. The first joint proposal within D
meters of the ground truth position is taken as a true positive, while other proposals also within D meters count as
false positives. This penalizes multiple spurious detections
near the correct position, which might slow a downstream
3. 4. Joint position proposals
Body part recognition as described above infers per-pixel
information. This information must now be pooled across
pixels to generate reliable proposals for the positions of 3D
skeletal joints. These proposals are the final output of our
algorithm and could be used by a tracking algorithm to self-initialize and recover from failure.
A simple option is to accumulate the global 3D centers of
probability mass for each part, using the known calibrated
depth. However, outlying pixels severely degrade the quality
of such a global estimate. We consider two algorithms: a
fast algorithm based on simple bottom-up clustering and a
more accurate algorithm based on mean shift, which shall
now be described.
We employ a local mode-finding approach based on
mean shift6 with a weighted Gaussian kernel. We define a
density estimator per body part as
where ◯ is a coordinate in 3D space, N is the number of
image pixels, wic is a pixel weighting, ◯i is the reprojection of
image pixel xi into world space given depth dI(xi), and bc is a
learned per-part bandwidth. The pixel weighting wic considers both the inferred body part probability at the pixel and
the world surface area of the pixel:
wic = P(c|I, xi) · dI(xi) 2.
This ensures that density estimates are depth invariant and
give a small but significant improvement in joint prediction
accuracy. Depending on the definition of body parts, the
posterior P(c|I, x) can be pre-accumulated over a small set of
parts. For example, in our experiments the four body parts
covering the head are merged to localize the head joint.