Occlusion Handling for the
Integration of Virtual Objects into Video

VISAPP 2012 - International Conference on Computer Vision Theory and Applications


This paper demonstrates how to effectively exploit occlusion and reappearance information of feature points in structure and motion recovery from video. Due to temporary occlusion with foreground objects, feature tracks discontinue. If these features reappear after their occlusion, they are connected to the correct previously discontinued trajectory during sequential camera and scene estimation. The combination of optical flow for features in consecutive frames and SIFT matching for the wide baseline feature connection provides accurate and stable feature tracking.
The knowledge of occluded parts of a connected feature track is used to feed a segmentation algorithm which crops the foreground image regions automatically. The resulting segmentation provides an important step in scene understanding which eases integration of virtual objects into video significantly. The presented approach enables the automatic occlusion of integrated virtual objects with foreground regions of the video.
Demonstrations show very realistic results in augmented reality.

Playground Sequence: download (120 MB)

  • Input
  • Result


  • Video download (24 MB):

  • VISAPP2012.avi
  • Presentation:

  • Full Paper presentation at VISAPP 2012 pdf
  • Selected to appear as Extended and Revised Version in:

    Communications in Computer and Information Science (CCIS), Springer
    Computer Vision, Imaging and Computer Graphics. Theory and Applications International Joint Conference, VISIGRAPP 2012,
    Revised Selected Papers

    Related Work:

    Feature Trajectory Retrieval with Application to Accurate Structure and Motion Recovery