TNT logo LUH TNT

MPI08_Database

TNT members involved in this project:
Prof. Dr.-Ing. Bodo Rosenhahn
Show all

Multisensor-Fusion for 3D Full-Body Human Motion Capture

Gerard Pons-Moll, Andreas Baak, Thomas Helten, Meinard Müller, Hans-Peter Seidel, and Bodo Rosenhahn


Abstract


In this work, we present an approach to fuse video with orientation data obtained from extended inertial sensors to improve and stabilize full-body human motion capture. Even though video data is a strong cue for motion analysis, tracking artifacts occur frequently due to ambiguities in the images, rapid motions, occlusions or noise. As a complementary data source, inertial sensors allow for drift- free estimation of limb orientations even under fast motions. However, accurate position information cannot be obtained in continuous operation. Therefore, we propose a hybrid tracker that combines video with a small number of inertial units to compensate for the drawbacks of each sensor type: on the one hand, we obtain drift-free and accurate position information from video data and, on the other hand, we obtain accurate limb orientations and good performance under fast motions from inertial sensors. In several experiments we demonstrate the increased performance and stability of our human motion tracker.

Data


The indoor motion capture dataset (MPI08) used in the CVPR 2010 paper is freely available for your own tests and experiments. The data is only for research purposes. If you use this data, please acknowledge the effort that went into data collection by citing the corresponding papers Multisensor Fusion for 3D Full-Body motion capture pdfBibTeX and Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors pdf BibTeX

The dataset consists of:
  • sequences : multi-view sequences obtained from 8 calibrated cameras.
  • silhouettes : binary segmented images obtained with chroma-keying.
  • meshes : 3D laser scans for each of the four actors in the dataset and also the registered meshes with inserted skeletton.
  • projection matrices : one for each of the 8 cameras.
  • orientation data : raw and calibrated and sensor orientation data (5 sensors)

3D scans

Multiview sequences

5 sensors



The full dataset can be downloaded directly from here. The sequences for each of the four actors ab,hb,br,mm, are included as compressed h.264 with high quality. Additionally, we include the segmented silhouettes for all the database. The silhouettes are run-length encoded (loss-less) for efficiency.
This matlab demo script shows how to read and use the data in MPI08.It loads a mesh and a sequence and plots the 3D model with the sensor orientations in the first frame and the projection of the model into the first frame.

ab br hb mm
sequences ab.tar.gz br.tar.gz hb.tar.gz mm.tar.gz
silhouettes MPI08_silhouettes.tar.gz
meshes
and
projection matrices
InputFiles.tar.gz
orientation data MPI08_PriorFiles.tar.gz
documents Documents.tar.gz

Note: The videos are compressed with high quality, however if you would like to obtain the sequences as lossless png images please contact pons@tnt.uni-hannover.de

Example sequences in the database (Click on the images to see the multi-view sequences ! )

  Cartwheel
  Jumping jack and skiing      
  Rotating both arms      
  Kicking

Video


no movie


References


Gerard Pons-Moll, Andreas Baak, Thomas Helten, Meinard Müller, Hans-Peter Seidel, and Bodo Rosenhahn
Multisensor-Fusion for 3D Full-Body Human Motion Capture
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010
pdf BibTeX
Andreas Baak, Thomas Helten, Gerard Pons-Moll, Meinard Müller, Hans-Peter Seidel, and Bodo Rosenhahn
Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors
European Conference on Computer Vision (ECCV Workshops), September 2010
pdf BibTeX