Humans

Mitarbeiter: Jörn Ostermann, Bodo Rosenhahn, Roberto Henschel, Petrissa Zell, Bastian Wandt, Felix Kuhnke, Maren Awiszus, Marco Rudolph, Lars Rumberg, Yuren Cong
Introduction

The detection of human motion has numerous applications in areas such as film production, animation, medical analysis, sports science and natural human-computer interaction. Traditionally, systems are used to track the 3D position of markers glued to the body and reconstruct the body movement from this information. Such systems are expensive, difficult to install and the movements can only be recorded in a small recording area.

The Institute for Information Processing (TNT) is researching novel methods to reconstruct the motions in arbitrary environments and without special markers. Apart from standard cameras, inertial sensors worn on the body are used, as well. In addition to recording the movements, methods are being researched to estimate the underlying forces and moments in the musculoskeletal system of the human body.

Current research topics

Monocular camera:

In a project for the acquisition of human motion based on a single moving camera, optimization methods are being developed which reconstruct not only posture but also camera movement and additionally estimate anthropometric variables of the actor. This enables the automatic recording of human motion with a standard camera, as found e.g. in smartphones.

Inertial sensors and sensor fusion:

In addition to camera-based approaches, reconstruction methods of human motion using inertial sensors are being researched. The sensors are worn on the body underneath the clothing and thus allow easy recording of movement in everyday situations and are suitable for long-term recordings [2]. In addition, fusion approaches are being developed to compensate for sensor uncertainties by combining them with image information from video data. This increases the accuracy and robustness of motion reconstruction.

Physical modeling:

A further research focus is the physical modeling and analysis of the recorded motion. The used methods include both, traditional approaches, such as forward and inverse dynamics, and machine learning approaches. The reconstruction of the acting forces and moments can be used, for example, to assess movements with respect to their efficiency or load level.

Methodologies used

Body:

pose estimation, body shape models, subspace projections, auto encoder, physical models, deep learning, generative adversarial networks, sensor fusion, IMUs

Faces:

Face detection, emotion recognition, extraction of distinctive features, 3D face reconstruction, processing of 3D meshes, motion capture, pose estimation, visual speech synthesis, virtual avatars

References
  • Conference Contributions
    • Felix Kuhnke, Lars Rumberg, Jörn Ostermann
      Two-Stream Aural-Visual Affect Analysis in the Wild
      15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE Computer Society, pp. 366-371, May 2020
    • Petrissa Zell, Bodo Rosenhahn, Bastian Wandt
      Weakly-supervised Learning of Human Dynamics
      European Conference on Computer Vision (ECCV), August 2020
    • Andrea Hornakova*, Roberto Henschel*, Bodo Rosenhahn, Paul Swoboda, (* equal contribution)
      Lifted Disjoint Paths with Application in Multiple Object Tracking
      Proceedings of the 37th International Conference on Machine Learning (ICML), July 2020
    • Sami Brandt, Hanno Ackermann, Stella Graßhof
      Uncalibrated Non-Rigid Factorisation by Independent Subspace Analysis
      Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Seoul, Korea, October 2019
    • Roberto Henschel, Timo von Marcard, Rosenhahn Bodo
      Simultaneous Identification and Tracking of Multiple People using Video and IMUs
      Computer Vision and Pattern Recognition Workshops (CVPRW), June 2019
    • Roberto Henschel, Yunzhe Zou, Bodo Rosenhahn
      Multiple People Tracking using Body and Joint Detections
      Computer Vision and Pattern Recognition Workshops (CVPRW), June 2019
    • Bastian Wandt, Bodo Rosenhahn
      RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation
      Computer Vision and Pattern Recognition (CVPR), IEEE, June 2019
    • Maren Awiszus, Stella Graßhof, Felix Kuhnke, Jörn Ostermann
      Unsupervised Features for Facial Expression Intensity Estimation over Time
      Computer Vision and Pattern Recognition Workshops (CVPRW), June 2018
    • Felix Kuhnke
      Head Pose Estimation using Convolutional Neural Networks
      Proceedings of the 4th Summer School on Video Compression and Processing (SVCP) 2018, Leibniz Universität Hannover, Institut für Informationsverarbeitung, July 2018, edited by Voges, Jan
    • Roberto Henschel, Laura Leal-Taixé, Daniel Cremers, Bodo Rosenhahn
      Fusion of Head and Full-Body Detectors for Multi-Object Tracking
      Computer Vision and Pattern Recognition Workshops (CVPRW), accepted as spotlight presentation, June 2018
    • Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, Gerard Pons-Moll
      Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera
      European Conference on Computer Vision, September 2018
    • Bastian Wandt, Hanno Ackermann, Bodo Rosenhahn
      A Kinematic Chain Space for Monocular Motion Capture
      ECCV Workshops, September 2018
    • Stella Graßhof, Hanno Ackermann, Sami Brandt, Jörn Ostermann
      Apathy is the Root of all Expressions
      12th IEEE Conference on Automatic Face and Gesture Recognition (FG2017), Washington D.C., USA, 2017
    • Stella Graßhof, Hanno Ackermann, Felix Kuhnke, Jörn Ostermann, Sami Brandt
      Projective Structure from Facial Motion
      15th IAPR International Conference on Machine Vision Applications (MVA) (accepted), Nagoya (Japan), May 2017
    • Felix Kuhnke, Jörn Ostermann
      Visual Speech Synthesis From 3D Mesh Sequences Driven By Combined Speech Features
      Proc. of the IEEE International Conference on Multimedia and Expo (ICME), IEEE, Hong Kong, July 2017
    • Petrissa Zell, Bastian Wandt, Bodo Rosenhahn
      Joint 3D Human Motion Capture and Physical Analysis from Monocular Videos
      The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017
    • Petrissa Zell, Bodo Rosenhahn
      Learning-Based Inverse Dynamics of Human Motion
      The IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 842-850, October 2017
    • Thiemo Alldieck, Marc Kassubeck, Bastian Wandt, Bodo Rosenhahn, Marcus Magnor
      Optical Flow-based 3D Human Motion Estimation from Monocular Video
      German Conference on Pattern Recognition (GCPR), September 2017
    • Holger Meuel, Luis Angerstein, Roberto Henschel, Bodo Rosenhahn, Jörn Ostermann
      Moving Object Tracking for Aerial Video Coding using Linear Motion Prediction and Block Matching
      Proceedings of the 32nd Picture Coding Symposium (PCS), pp. 1-5, Nuremberg, Germany, December 2016
    • Stella Graßhof, Hanno Ackermann, Jörn Ostermann
      Estimation of Face Parameters using Correlation Analysis and a Topology Preserving Prior
      14th IAPR International Conference on Machine Vision Applications (MVA), Tokyo, May 2015
    • Karsten Vogt, Oliver Müller, Jörn Ostermann
      Facial Landmark Localization using Robust Relationship Priors and Approximative Gibbs Sampling
      Advances in Visual Computing , Springer, Vol. 9475, pp. 365 -- 376, Las Vegas, December 2015, edited by George Bebis et al.
    • Bastian Wandt, Hanno Ackermann, Bodo Rosenhahn
      3D Human Motion Capture from Monocular Image Sequences
      IEEE Conference on Computer Vision and Pattern Recognition Workshops, IEEE, June 2015
    • Roberto Henschel, Laura Leal-Taixé, Rosenhahn Bodo
      Solving Multiple People Tracking In A Minimum Cost Arborescence
      IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW), accepted as oral presentation, 1st Workshop on Benchmarking Multi-target Tracking (BMTT), Waikoloa Beach, Hawaii, USA, January 2015
    • Petrissa Zell and Bodo Rosenhahn
      A physics-based statistical model for human gait analysis
      German Conference on Pattern Recognition (GCPR), October 2015
    • Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, Silvio Savarese
      Learning an Image-based Motion Context for Multiple People Tracking
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, Ohio, USA, June 2014
    • Roberto Henschel, Laura Leal-Taixé, Bodo Rosenhahn
      Efficient Multiple People Tracking Using Minimum Cost Arborescences
      German Conference on Pattern Recognition (GCPR), accepted as oral presentation, Münster, Germany, September 2014
    • Alina Kuznetsova, Bodo Rosenhahn
      On calibration of a low-cost time-of-flight camera
      IEEE European Conference on Computer Vision Workshops (ECCVW), 4rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV), September 2014
    • Stella Graßhof, Jörn Ostermann
      Performance of Image Registration and Its Extensions for Interpolation of Facial Motion
      PSIVT 2013 Workshops, Springer Lecture Notes on Computer Sciences (LNCS), pp. 216--227, October 2013
    • Gerard Pons-Moll+, Jonathan Taylor+, Jamie Shotton, Aaron Hertzmann, Andrew Fitzgibbon
      Metric Regression Forests for Human Pose Estimation
      British Machine Vision Conference ( BMVC ) (+ dennotes equal contribution)
      Best Science Paper Award, September 2013
    • Alina Kuznetsova, Bodo Rosenhahn
      Hand pose estimation from a single RGB-D image
      9th International Symposium on Visual Computing (ISVC), July 2013
    • Alina Kuznetsova, Laura Leal-Taixé, Bodo Rosenhahn
      Real-time sign language recognition using a consumer depth camera
      IEEE International Conference on Computer Vision Workshops (ICCVW), 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV), December 2013
    • Laura Leal-Taixé, Gerard Pons-Moll, Bodo Rosenhahn
      Branch-and-price global optimization for multi-view multi-object tracking
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Providence, Rhode Island, USA., June 2012
    • Kang Liu, Joern Ostermann
      Realistic Head Motion Synthesis for an Image-based Talking Head
      FG 2011, The 9th IEEE Conference on Automatic Face and Gesture Recognition , p. 6, Santa Barbara, CA, March 2011
    • Kang Liu, Joern Ostermann
      Realistic Facial Expression Synthesis for an Image-based Talking Head
      IEEE Conference on Multimedia and Expo, ICME2011 , p. 6, Barcelona, Spain, July 2011
    • Kang Liu, Joern Ostermann
      Evaluation of an Image-based Talking Head with Realistic Facial Expression and Head Motion
      Proceedings of CASA (Computer Animation and Social Agents) workshop on Emotion-based Interaction, Chengdu, China, May 2011
    • Gerard Pons-Moll, Andreas Baak, Juergen Gall, Laura Leal-Taixe, Meinard Mueller, Hans-Peter Seidel, Bodo Rosenhahn
      Outdoor Human Motion Capture using Inverse Kinematics and von Mises-Fisher Sampling
      IEEE International Conference on Computer Vision (ICCV), November 2011
    • Laura Leal-Taixé, Gerard Pons-Moll, Bodo Rosenhahn
      Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker
      IEEE International Conference on Computer Vision Workshops (ICCVW). 1st Workshop on Modeling, Simulation and Visual Analysis of Large Crowds, November 2011
    • Kang Liu, Joern Ostermann
      Image-based Talking Head: Analysis and Synthesis
      DAGA 2010, 36. International Conference on Acoustics, Deutschen Gesellschaft für Akustik, pp. 87-88, Berlin, March 2010
    • Nils Hasler, Thorsten Thormählen, Bodo Rosenhahn, Hans-Peter Seidel
      Learning Skeletons for Shape and Pose
      ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Washington , February 2010
    • Gerard Pons-Moll, Andreas Baak, Thomas Helten, Meinard Müller, Hans-Peter Seidel, Bodo Rosenhahn
      Multisensor-Fusion for 3D Full-Body Human Motion Capture
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010
    • Andreas Baak, Thomas Helten, Meinard Müller, Gerard Pons-Moll, Bodo Rosenhahn, Hans-Peter Seidel
      Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors
      European Conference on Computer Vision (ECCV Workshops), September 2010
    • Axel Weissenfeld, Kang Liu, Joern Ostermann
      Video-Realistic Image-based Eye Animation System
      EUROGRAPHICS 2009 (Short Paper), Munich, April 2009
    • Kang Liu, Joern Ostermann
      Minimized Database of Unit Selection in Visual Speech Synthesis Without Loss of Naturalness
      The 13th International Conference on Computer Analysis of Images and Patterns CAIP2009, Springer-Verlag Berlin Heidelberg, pp. 1212-1219, Münster, Germany, September 2009, edited by X. Jiang and N. Petkov
    • Kang Liu, Joern Ostermann
      An Image-based Talking Head System
      LIPS 2009 Special Session in AVSP 2009, Norwich, UK, September 2009
    • Nils Hasler, Bodo Rosenhahn, Thorsten Thormählen, Michael Wand, Hans-Peter Seidel
      Markerless Motion Capture with Unsynchronized Moving Cameras
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, USA, 2009
    • Nils Hasler, Carsten Stoll, Bodo Rosenhahn, Thorsten Thormählen, H.-P. Seidel
      Estimating Body Shape of Dressed Humans
      Shape Modeling International, Beijing, 2009
    • Gerard Pons-Moll, Bodo Rosenhahn
      Ball Joints for Marker-less Human Motion Capture
      IEEE Workshop on Applications of Computer Vision (WACV), Snow Bird, Utah, USA, December 2009
    • Kang Liu, Axel Weissenfeld, Joern Ostermann, Xinghan Luo
      Robust AAM Building for Morphing in an Image-based Facial Animation System
      IEEE Multimedia and Expo, 2008 IEEE International Conference on , Hannover, Germany, June 2008
    • Kang Liu, Joern Ostermann
      Realistic Facial Animation System for Interactive Services
      Interspeech 2008, LIPS 2008: Visual Speech Synthesis Challenge, Brisbane, Australia, September 2008
    • Kang Liu, Joern Ostermann
      Realistic Talking Head for Human-Car-Entertainment Services
      IMA 2008 Informationssysteme für mobile Anwendungen, GZVB e.V. (Hrsg.), pp. 108-118, Braunschweig, Germany, September 2008
    • B. Rosenhahn, C. Schmaltz, T. Brox, J. Weickert, D. Cremers, H.-P. Seidel
      Markerless Motion Capture of Man-Machine Interaction
      IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, 2008
    • J. Gall, B. Rosenhahn, H.-P. Seidel
      Drift-free Tracking of Rigid and Articulated Objects
      IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, 2008
    • B. Rosenhahn, T. Brox, H.-P. Seidel
      Scaled Motion Dynamics for Markerless Motion Capture
      IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, Minnesota, USA., 2007
    • T. Brox, B. Rosenhahn, D. Cremers, H.-P. Seidel
      Nonparametric Density Estimation with Adaptive Anisotropic Kemels for Human Motion Tracking
      2nd. Workshop on Human Motion, Springer-Verlag, Berlin Heidelberg, pp. 152-165, 2007, edited by Elgammal, A.; Rosenhahn, B. ; Klette, R.
    • Kang Liu, Axel Weissenfeld, Joern Ostermann
      Parameterization of Mouth Images by LLE and PCA for Image-based Facial Animation
      ICASSP06,Toulouse, France IEEE Proceedings, IEEE, Vol. 5, pp. 461-464, May 2006
    • Axel Weissenfeld, Onay Urfalioglu, Kang Liu, Joern Ostermann
      Robust Rigid Head Motion Estimation based on Differential Evolution
      IEEE International Conference on Multimedia & Expo 2006, IEEE Multimedia and Expo, 2006 IEEE International Conference on, pp. 225 - 228, Toronto, CN, July 2006
    • Axel Weissenfeld, Kang Liu, Wei Liu, Joern Ostermann
      Image-based Head Animation System
      1. Kongress Multimediatechnik 2006, Institut für Multimediatechnik GmbH -IFM, pp. 67-72, Wismar, November 2006
    • T. Brox, B. Rosenhahn, U. Kersting, D. Cremers
      Nonparametric Density Estimation for Human Tracking
      Pattern Recognition 2006, DAGM, Springer-Verlag, Berlin Heidelberg, pp. 546-555, Berlin, 2006, edited by Franke, K.; Mueller, R.;Nickolay, B.; Schaefer, R.
    • Axel Weissenfeld, Kang Liu, Sven Klomp, Joern Ostermann
      Personalized Unit Selection for an Image-based Facial Animation System
      IEEE MMSP 2005, Shanghai/China, IEEE, November 2005
    • Jörn Ostermann, Axel Weissenfeld, Kang Liu
      Talking Faces - Technologies and Applications (Keynote)
      Vision, Video, and Graphics 2005, Eurographics Association, pp. 157-158, University of Edinburgh, July 2005, edited by Emanuele Trucco
    • A.C. Andres del Valle, Joern Ostermann
      3D talking head customization by adapting a generic model to one uncalibrated picture
      ISCAS 2001, Sydney, Australia, Vol. 2, pp. 325-328, May 2001
    • Joern Ostermann, D. Millen
      Talking heads and synthetic speech: An architecture for supporting electronic commerce
      ICME 2000, International Conference on Multimedia and Expo, New York, USA, IEEE CNF, Vol. 1, pp. 71-74, July 2000
    • Joern Ostermann, Y. Wang, M. Beutnagel, A. Fischer
      Integration of talking heads and text-to-speech synthesizers for visual TTS
      International Conference on Spoken Language Processing, Sydney, Australia, pp. 297-300, December 1998
  • Journals
    • Petrissa Zell, Bodo Rosenhahn
      Learning inverse dynamics for human locomotion analysis
      Neural Computing and Applications, Springer Nature, December 2019
    • Timo von Marcard, Bodo Rosenhahn, Michael Black, Gerard Pons-Moll
      Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs
      Computer Graphics Forum 36(2), Proceedings of the 38th Annual Conference of the European Association for Computer Graphics (Eurographics), 2017
    • Bastian Wandt, Hanno Ackermann, Bodo Rosenhahn
      3D Reconstruction of Human Motion from Monocular Image Sequences
      Transactions on Pattern Analysis and Machine Intelligence, IEEE, Vol. 38, No. 8, pp. 1505-1516, 2016
    • Timo von Marcard, Gerard Pons-Moll, Bodo Rosenhahn
      Human Pose Estimation from Video and IMUs
      Transactions on Pattern Analysis and Machine Intelligence, IEEE, Vol. 38, No. 8, pp. 1533-1547, January 2016
    • Kang Liu, Joern Ostermann
      Evaluation of an Image-based Talking Head with Realistic Facial Expression and Head Motion
      Journal on Multimodal User Interfaces, Special issue: Emotion-based Interaction, October 2011
    • Kang Liu, Joern Ostermann
      Optimization of An Image-based Talking Head System
      Special issue on animating virtual speakers or singers from audio: Lip-synching facial animation, EURASIP Journal on Audio, Speech, and Music Processing, Hindawi Publishing Corporation, Vol. 2009, September 2009
    • Axel Weissenfeld, Kang Liu, Jörn Ostermann
      Video-realistic image-based eye animation via statistically driven state machines
      The Visual Computer, Springer Berlin / Heidelberg, November 2009
    • Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, Seidel Hans-Peter
      A Statistical Model of Human Pose and Body Shape
      Computer Graphics Forum (Proc. Eurographics 2009), Munich, Germany, 2009
    • B. Rosenhahn, U. Kersting, K. Powell, R. Klette, G. Klette, H.-P. Seidel
      A system for articulated tracking incorporating a clothing model
      Machine Vision and Applications, Springer Verlag, Berlin-Heidelberg, Vol. 18, No. 1, pp. 25-40, February 2007
    • Axel Weissenfeld, Kang Liu, Joern Ostermann
      Gesichtsanimation mit Image-based Rendering für Dialogsysteme
      Telekommunikation Aktuell, Berichte aus Forschung und Entwicklung in Informationstechnik und Telekommunikation, 60. Jahrgang, Heft 07-12, Verlag für Wissenschaft und Leben, Erlangen, December 2006
    • Jörn Ostermann, Lawrence S. Chen, Thomas S. Huang
      Animated Talking Head with Personalized 3D Head Model
      VLSI Signal Processing, Kluwer Academic Publishers, The Netherlands, pp. 97-105, 1998
    • Joern Ostermann
      Animated Talking Head with Personalized 3D Head Model
      Journal of VLSI Signal Processing, Kluwer Academic Publishers, p. 9, 1998, edited by Chen, Lawrence S.; Huang, Thomas S.
  • Book Chapters
    • Petrissa Zell, Bastian Wandt, Bodo Rosenhahn
      Physics-based Models for Human Gait Analysis
      Handbook of Human Motion, Springer International Publishing, 2018, edited by Bertram Müller, Sebastian I. Wolf
    • Laura Leal-Taixé, Gerard Pons-Moll, Bodo Rosenhahn
      Exploiting pedestrian interaction via global optimization and social behaviors
      Theoretic Foundations of Computer Vision: Outdoor and Large-Scale Real-World Scene Analysis, Springer, April 2012, edited by F. Dellaert, J.-M. Frahm, M. Pollefeys, L. Leal-Taixé, B. Rosenhahn
    • Laura Leal-Taixé, Bodo Rosenhahn
      Pedestrian interaction in tracking: the social force model and global optimization methods
      Modeling, Simulation and Visual Analysis of Crowds: A multidisciplinary perspective, Springer, September 2012, edited by Saad Ali, Ko Nishino, Dinesh Manocha and Mubarak Shah
    • B. Rosenhahn, Uwe G. Kersting, K. Powell, T. Brox, Hans-Peter Seidel
      Tracking Clothed People
      Human Motion - Understanding, Modelling, Capture and Animation, Springer Verlag, Dordrecht, The Netherlands, Vol. 36, pp. 295-317, 2007, edited by Rosenhahn B.; Klette R.; Metaxas D.
  • Technical Report
    • Felix Kuhnke, Stella Graßhof, Jörn Ostermann
      Das Gesicht als Interface zwischen Mensch und Maschine - Wie wir zukünftig mit Robotern kommunizieren
      Unimagazin - Forschungsmagazin der Leibniz Universität Hannover, pp. 14-16, Hannover, 2016
    • Roberto Henschel, Laura Leal-Taixé, Bodo Rosenhahn, Konrad Schindler
      Tracking with multi-level features
      arXiv, July 2016
    • Joern Ostermann, Erich Haratsch
      Parameter-Based Model-Independent Animation of Personalized Talking Heads
      IEEE Transactions on circuits and systems for video technology, IEEE Transactions on circuits and systems for video technology, p. 24, 1996