TNT logo LUH TNT

3D Reconstruction and Camera Motion Estimation

TNT members involved in this project:
Hanno Ackermann, Ph.D.
Dr.-Ing. Marco Munderloh
Prof. Dr.-Ing. Bodo Rosenhahn
Show all

When a picture is taken by a camera, the depth of the observed objects is lost. 3D-reconstruction is to recover the missing 3D information of the scene, and to estimate position and orientation of all cameras at the same time as shown in the following example:


This an important step in applications like augmented reality (e.g. integrating virtual objects into real image sequences), movie production, robot or car navigation, or reconstruction of historical buildings or other architectural purposes.
Or for reconstructing Cuxi.

We extract relevant information such as feature points or line structures from one or more input images. Given this information, we wish to estimate the positions and orientations of the cameras which observed the images, and the positions of the features within the 3D-scene (cf. to the image).


We pursue several different approaches in this institute: a traditional method is to estimate a tentative reconstruction from very few images (epipolar geometry), and initialize a nonlinear optimization with this input (bundle adjustment).
A different algorithm is the so-called factorization method. While it converges robustly, such an estimation is susceptible to missing data. In our group, we investigate the mathematical structure of this problem. We were able to develop surprisingly simple solutions which also apply to more general problems such as PCA.


Figure: Detected feature points, trajectories, reconstructed scene and camera path, and the application of integrating a virtual object ( Click on the images for some example videos ! )

References

Occlusion Handling by Retrieval of Discontinued Trajectories
[1] Kai Cordes: Occlusion Handling in Scene Reconstruction from Video, Dissertation, VDI, Vol. 10, No. 834, p. 118, 2014
[2] Kai Cordes, Björn Scheuermann, Bodo Rosenhahn, Jörn Ostermann: Learning Object Appearance from Occlusions using Structure and Motion Recovery, ACCV, Springer, 2012
[3] Project Pages: ISVC'11, VISAPP'12, ACCV'12
Highly-Accurate and High-Resolution Feature Evaluation Benchmarks
- Benchmark data: SSCI'11, CAIP'13, Project Page
- Example for accuracy evaluation: AVSS'14
A Linear Solution to 1-Dimensional Subspace Fitting under Incomplete Data
(ACCV'10)
Multilinear Pose and Body Shape Estimation of Dressed Subjects from Image Sets
(CVPR'10)
Scale Invariant Feature Detection based on Shape Models:
[1] Kai Cordes, Oliver Müller, Bodo Rosenhahn, Jörn Ostermann: HALF-SIFT: High-Accurate Localized Features for SIFT, CVPRw, IEEE, 2009
[2] Kai Cordes, Oliver Müller, Bodo Rosenhahn, Jörn Ostermann: Bivariate Feature Localization for SIFT Assuming a Gaussian Feature Shape, ISVC, Springer, 2010
[3] Kai Cordes, Oliver Topic, Manuel Scherer, Carsten Klempt, Bodo Rosenhahn, Jörn Ostermann: Classification of Atomic Density Distributions using Scale Invariant Blob Localization, ICIAR, Springer, 2011
[4] Kai Cordes, Jörn Ostermann: Increasing the Precision of Junction Shaped Features, International Conference on Machine Vision Applications (MVA), IEEE, pp. 295--298, Tokyo, Japan, May 2015
Affine Structure-from-Motion by Global and Local Constraints
(CVPR'09)
Robust Registration of 3D-Point Data and a Triangle Mesh:
[1] Onay Urfalioglu, Patrick Mikulastik, Ivo Stegmann: Scale Invariant Robust Registration of 3D-Point Data and a Triangle Mesh by Global Optimization, ACIVS, Springer, 2006
[2] Kai Cordes, Patrick Mikulastik, Alexander Vais, Jörn Ostermann: Extrinsic Calibration of a Stereo Camera System using a 3D CAD Model considering the Uncertainties of Estimated Feature Points, CVMP, IEEE, 2009
Keyframe Selection for Camera Motion and Structure Estimation
(ECCV'04)
User-Friendly Integration of Virtual Objects into Image Sequences with Mosaics
(VI'03)
Robust Estimation of Camera Parameters for Integration of Virtual Objects into Video Sequences
(IWSNHC3DI'99 (.ps))

Show recent publications only
  • Hanno Ackermann, Bodo Rosenhahn
    Non-Rigid Self-Calibration Of A Projective Camera
    Proceedings of the 11th Asian Conference on Computer Vision (ACCV), November 2012
  • Books
    • Kai Cordes
      Occlusion Handling in Scene Reconstruction from Video
      VDI Verlag, Vol. 10, No. 834, p. 118, Düsseldorf, 2014
  • Book Chapters
    • Kai Cordes, Björn Scheuermann, Bodo Rosenhahn, Jörn Ostermann
      Foreground Segmentation from Occlusions using Structure and Motion Recovery
      Computer Vision, Imaging and Computer Graphics. Theory and Applications, Communications in Computer and Information Science (CCIS), Springer Berlin/Heidelberg, Vol. 359, pp. 340--353, May 2013, edited by Gabriela Csurka et al.
  • Journals
    • Hanno Ackermann, Kenichi Kanatani
      Fast projective reconstruction: Toward ultimate efficiency
      IPSJ Transactions on Computer Vision and Image Media, Vol. 49, pp. 68-78, March 2008
    • Kai Cordes, Jörn Ostermann
      Integration virtueller Objekte in natürliche Bildsequenzen
      Telekommunikation Aktuell, Berichte aus Forschung und Entwicklung in Informationstechnik und Telekommunikation, 60. Jahrgang, Heft 07-12, Verlag für Wissenschaft und Leben, Erlangen, December 2006
    • Thorsten Thormählen, Hellward Broszio
      Automatic line-based estimation of radial lens distortion
      Integrated Computer-Aided Engineering, IOS Press, Vol. 12, No. 2, pp. 177 - 190, April 2005
    • Hellward Broszio, Thorsten Thormählen, Patrick Mikulastik
      Using Mosaic Images to Simplify the Integration of Virtual Objects into Image Sequences
      International Journal of Robotics and Automation, ACTA Press , Vol. 20, No. 1, pp. 34-41, 2005
  • Conference Contributions
    • Kai Cordes, Lukas Grundmann, Jörn Ostermann
      Feature Evaluation with High-Resolution Images
      The 16th International Conference on Computer Analysis of Images and Patterns (CAIP), Lecture Notes in Computer Science (LNCS), Springer International Publishing Switzerland, Vol. 9256, pp. 374--386, Valletta, Malta, September 2015, edited by George Azzopardi and Nicolai Petkov
    • Kai Cordes, Mark Hockner, Hanno Ackermann, Bodo Rosenhahn, Jörn Ostermann
      WM-SBA: Weighted Multibody Sparse Bundle Adjustment
      The 14th IAPR International Conference on Machine Vision Applications (MVA), pp. 162--165, Tokyo, Japan, May 2015
    • Kai Cordes, Jörn Ostermann
      Increasing the Precision of Junction Shaped Features
      The 14th IAPR International Conference on Machine Vision Applications (MVA), pp. 295--298, Tokyo, Japan, May 2015
    • Kai Cordes, Bodo Rosenhahn, Jörn Ostermann
      Localization Accuracy of Interest Point Detectors with Different Scale Space Representations
      The 11th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS), IEEE Computer Society, pp. 247--252, Seoul, Korea, August 2014
    • Christian Cordes, Hanno Ackermann, Bodo Rosenhahn
      A Low-Rank Constraint for Parallel Stereo Cameras
      Proceedings of the German Conference on Pattern Recognition (GCPR), Springer Lecture Notes on Computer Sciences (LNCS), pp. 31-40, September 2013, edited by Joachim Weickert, Matthias Hein, Bernt Schiele
    • Kai Cordes, Bodo Rosenhahn, Jörn Ostermann
      High-Resolution Feature Evaluation Benchmark
      The 15th International Conference on Computer Analysis of Images and Patterns (CAIP), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 8047, pp. 327-334, York, UK, August 2013, edited by Richard Wilson et al.
    • Kai Cordes, Björn Scheuermann, Bodo Rosenhahn, Jörn Ostermann
      Learning Object Appearance from Occlusions using Structure and Motion Recovery
      The 11th Asian Conference on Computer Vision (ACCV), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 7726, pp. 611--623, Daejeon, Korea, November 2012, edited by Kyoung Mu Lee, Jim Rehg, Yasuyuki Matsushita, and Zhanyi Hu
    • Kai Cordes, Björn Scheuermann, Bodo Rosenhahn, Jörn Ostermann
      Occlusion Handling for the Integration of Virtual Objects into Video
      The 7th International Conference on Computer Vision Theory and Applications (VISAPP), SciTePress, pp. 173--180, Rome, Italy, February 2012, edited by Gabriela Csurka and José Braz
    • H. Ackermann, B. Rosenhahn
      Projective Reconstruction from Incomplete Trajectories by Global and Local Constraints
      The 8th European Conference on Visual Media Production (CVMP), November 2011
    • F. R. Schmidt, H. Ackermann, B. Rosenhahn
      Multilinear Model Estimation with L2-Regularization
      33rd Annual Symposium of the German Association for Pattern Recognition (DAGM) , September 2011
    • Kai Cordes, Oliver Müller, Bodo Rosenhahn, Jörn Ostermann
      Feature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery
      Advances in Visual Computing, 7th International Symposium (ISVC), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 6938, pp. 156--167, Las Vegas, NV, USA, September 2011, edited by George Bebis et al.
    • Kai Cordes, Oliver Topic, Manuel Scherer, Carsten Klempt, Bodo Rosenhahn, Jörn Ostermann
      Classification of Atomic Density Distributions using Scale Invariant Blob Localization
      Image Analysis and Recognition, 8th International Conference (ICIAR), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 6753, pp. 161--172, Burnaby, BC, Canada, June 2011, edited by Mohamed Kamel and Aurélio Campilho
    • Marco Munderloh, Holger Meuel, Joern Ostermann
      Mesh-based Global Motion Compensation for Robust Mosaicking and Detection of Moving Objects in Aerial Surveillance
      IEEE CVPR 2011, 1st Workshop of Aerial Video Processing (WAVP), pp. 1-6, Colorado Springs, Colorado, USA, June 2011
    • Kai Cordes, Bodo Rosenhahn, Jörn Ostermann
      Increasing the Accuracy of Feature Evaluation Benchmarks Using Differential Evolution
      IEEE Symposium Series on Computational Intelligence (SSCI) - IEEE Symposium on Differential Evolution (SDE), IEEE Computer Society, Paris, France, April 2011
    • Kai Cordes, Oliver Müller, Bodo Rosenhahn, Jörn Ostermann
      Bivariate Feature Localization for SIFT Assuming a Gaussian Feature Shape
      Advances in Visual Computing, 6th International Symposium (ISVC), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 6453, pp. 264--275, Las Vegas, NV, USA, November 2010, edited by George Bebis et al.
    • Hanno Ackermann, Bodo Rosenhahn
      A Linear Solution to 1-Dimensional Subspace Fitting under Incomplete Data
      Asian Conference on Computer Vision, Queenstown, New Zealand, November 2010
    • Nils Hasler, Hanno Ackermann, Bodo Rosenhahn, Thorsten Thormählen, Hans-Peter Seidel
      Multilinear Pose and Body Shape Estimation of Dressed Subjects from Image Sets
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, San Francisco, USA, June 2010
    • Kai Cordes, Patrick Mikulastik, Alexander Vais, Jörn Ostermann
      Extrinsic Calibration of a Stereo Camera System using a 3D CAD Model considering the Uncertainties of Estimated Feature Points
      The 6th European Conference on Visual Media Production (CVMP), IEEE Computer Society, pp. 135--143, London, UK, November 2009
    • Christian Kurz, Thorsten Thormählen, Bodo Rosenhahn, Hans-Peter Seidel
      Exploiting Mutual Camera Visibility in Multi-Camera Motion Estimation
      5th International Symposium on Visual Computing, Lecture Notes in Computer Science (LNCS), Springer, Las Vegas, USA, November 2009
    • Hanno Ackermann, Bodo Rosenhahn
      Trajectory Reconstruction for Affine Structure-from-Motion by Global and Local Constraints
      IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2009, IEEE Computer Society, pp. 2890-2897, Miami, USA, June 2009
    • Kai Cordes, Oliver Müller, Bodo Rosenhahn, Jörn Ostermann
      HALF-SIFT: High-Accurate Localized Features for SIFT
      The 22nd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Workshop on Feature Detectors and Descriptors: The State Of The Art and Beyond, IEEE Computer Society, pp. 31--38, Miami, FL, USA, June 2009
    • Kai Cordes, Patrick Mikulastik, Jörn Ostermann
      Calibration of a Stereo Camera System using a 3D CAD Calibration Object
      Zweiter Workshop für Optische Technologien (HOT), PZH, pp. 187--190, Hannover, November 2008, edited by Thomas Fahlbusch and Luca Spani Molella
    • Hanno Ackermann, Kenichi Kanatani
      Iterative Low Complexity Factorization for Projective Reconstruction
      2nd Workshop on Robot Vision, Springer, pp. 153-164, Auckland, New Zealand, February 2008, edited by Sommer, Gerald; Klette, Reinhard
    • Hanno Ackermann, Kenichi Kanatani
      Robust and efficient 3-D reconstruction by self-calibration
      IAPR Conference on Machine Vision Applications (MVA 2007), pp. 178-181, Tokyo, Japan, May 2007
    • Patrick Mikulastik, Hellward Broszio, Thorsten Thormaehlen, Onay Urfalioglu
      Error Analysis of Feature Based Disparity Estimation
      First Pacific Rim Symposium on Advances in Image and Video Technology, Springer LNCS 4319, pp. 1-12, December 2006
    • Patrick Mikulastik, Raphael Hoever, Onay Urfalioglu
      Error analysis of subpixel edge localisation
      Signal Processing for Image Enhancement and Multimedia Processing, Springer US, Vol. 31, No. 2, pp. 103-113, December 2006, edited by Ernesto Damiani, Kokou Yétongnon, Peter Schelkens, Albert Dipanda, Louis Legrand, Richard Chbeir
    • Onay Urfalioglu, Patrick Mikulastik, Ivo Stegmann
      Scale Invariant Robust Registration of 3D-Point Data and a Triangle Mesh by Global Optimization
      Proceedings of the 8th International Conference on Advanced Concepts for Intelligent Vision Systems (ACIVS 2006), Springer LNCS, Vol. 4179, pp. 1059-1070, Antwerp, Belgium, September 2006, edited by Jacques Blanc-Talon, Wilfried Philips, Dan C. Popescu, and Paul Scheunders
    • Onay Urfalioglu, Thorsten Thormaehlen, Hellward Broszio, Patrick Mikulastik
      Error Analysis of Camera Parameter Estimation based on Collinear Features
      Proceedings of the 3rd Canadian Conference on Computer and Robot Vision 2006 (CRV2006): Quebec/Canada, June 2006
    • Kenichi Kanatani, Yasuyuki Sugaya, Hanno Ackermann
      Uncalibrated factorization using a variable symmetric affine camera
      9th European Conference on Computer Vision (ECCV 2006), pp. 147-158, Graz, Austria, May 2006
    • Thorsten Thormählen, Hellward Broszio, Patrick Mikulastik
      Robust Linear Auto-calibration of a Moving Camera from Image Sequences
      Proceedings of the 7th Asian Conference on Computer Vision (ACCV), Hyderabad, India, Springer, LNCS 3852, Part II, ISSN 0302-9743, January 2006
    • Patrick Mikulastik
      Genauigkeitsanalyse von Kantenpunkten für Stereokamerasysteme
      in "Niccimon – Das Niedersächsische Kompetenzzentrum Informationssysteme für die mobile Nutzung"; Shaker Verlag Aachen 2006, ISBN 3-8322-5228-2, pp. 133-141, 2006
    • Onay Urfalioglu
      Robust Estimation with Non Linear Particle Swarm Optimization
      Proceedings of Mirage 2005 (Computer Vision / Computer Graphics Collaboration Techniques and Applications), INRIA Rocquencourt, France. March, 1-2 2005, 2005
    • Thorsten Thormählen, Hellward Broszio, Axel Weissenfeld
      Keyframe Selection for Camera Motion and Structure Estimation from Multiple Views
      Proceedings of the 8th European Conference on Computer Vision, Vol. 3021, No. 1, pp. 523-535, Prague, Czech Republic, May 2004, edited by Tomas Pajdla, Jirı Matas
    • Onay Urfalioglu
      Robust Estimation of Camera Rotation, Translation and Focal Length at High Outlier Rates
      Proceedings of Canadian Conference on Computer and Robot Vision, 2004
    • T. Thormählen, H. Broszio, I. Wassermann
      Robust Line-Based Calibration of Lens Distortion from a Single View
      Proceedings of Mirage 2003 ( Computer Vision / Computer Graphics Collaboration for Model-based Imaging, Rendering, Image Analysis and Graphical Special Effects), INRIA Rocquencourt, France, March, 10-12 2003, pp. 105-112, 2003
    • H. Broszio, T. Thormählen, P. Mikulastik
      User-Friendly Integration of Virtual Objects into Image Sequences with Mosaics
      Proceedings of the 16th International Conference on Vision Interface, Halifax, Canada, June, 11-13 2003, pp. 70-76, 2003
    • T. Thormählen, H. Broszio, P.N. Meier
      Three-Dimensional Endoscopy
      Falk Symposium No. 124, Medical Imaging in Gastroenterology and Hepatology, Hannover, Germany, September 2001, Kluwer Academic Publishers, 2002, ISBN 0-7923-8774-0, 2002
    • T. Thormählen, H. Broszio, P.N. Meier
      Automatische 3D-Rekonstruktion aus endoskopischen Bildfolgen
      Bildverarbeitung für die Medizin 2002, Leipzig, Germany, März 2002, Springer Verlag, 2002, ISBN 3-540-43225-6 , 2002
    • Y. Liu, T. Thormählen, H. Broszio
      Camera Motion Estimation for Object-Based Video Coding
      Recent Trends in Multimedia Information Processing, Proceedings of the 9th International Workshop on Systems, Signals and Image Processing, Manchester Town Hall, UK 7-8 November 2002, World Scientific, October 2002, ISBN 981-238-243-7 , 2002
    • Hellward Broszio, Oliver Grau
      Robust Estimation of Camera Parameters Pan, Tilt and Zoom for Integration of Virtual Objects into Video Sequences
      International Workshop on Synthetic-Natural Hybrid Coding and Three Dimensional Imaging, Santorini, Greece, September 15-17 1999, 1999