Highly-Accurate Performance Evaluation of Region Detectors

TNT members involved in this project:
Prof. Dr.-Ing. Jörn Ostermann
Show all

The detection of features is a fundamental step in many computer vision applications. Standing at the beginning of a processing pipeline, the accuracy of such an application is often determined by the accuracy of the detected features. Thus, the development and the evaluation of feature detectors is of high interest in the computer vision community.
Our evaluation is focused on affine invariant region detectors, e.g. (Cordes et al. 2010 ) :

The data set includes:

  • Image sequences showing planar scenes with changes in illumination and perspective.
  • Highly-accurate homography matrices H for each image pair. The matrices determine the mapping between the images. In order to provide fair results, the matrices are computed without using any feature detector.

For evaluation of region detectors

NEWS:


  • Projection of scene point P to camera planes:

  • Scene restricted to planar geometry
  • With homogeneous coordinates: H*p=p'


  • The data sets contain
    • image sequences in different resolutions (generated from the RAW data of a Canon EOS 350D camera)
    • homographies for the accurate mapping between image pairs of a sequence [1], [2]
  • Our data sets overcome the problem of limited accuracy of the data provided by Mikolajczyk et al. (cf. [1] for details)
  • Additionally, we provide
  • Easy to use and compare: use Matlab evaluation protocol (repeatability, matching score), with our ppm-images and homography-textfiles; evaluation examples: [4,5,6,7,8,9,11,12]
  • If you use the data for evaluation, please cite the corresponding paper(s)
  • Our data set has been included in the HBench challenge [10]


  • [1] Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: Increasing the Accuracy of Feature Evaluation Benchmarks Using Differential Evolution, Symposium on Differential Evolution (SDE), IEEE, 2011
  • [2] Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: High-Resolution Feature Evaluation Benchmark, Computer Analysis of Images and Patterns (CAIP), Springer, 2013
  • [3] Krystian Mikolajczyk, Cordelia Schmid: Scale & affine invariant interest point detectors. International Journal of Computer Vision (IJCV) 60, 63--86 (2004)
  • [4] Hossein Mobahi, C. Lawrence Zitnick, Yi Ma.: Seeing through the blur. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1736--1743 (2012)
  • [5] Kai Cordes, Bodo Rosenhahn, and Jörn Ostermann: Localization Accuracy of Interest Point Detectors with Different Scale Space Representations, Advanced Video and Signal-Based Surveillance (AVSS), pp. 247--252, IEEE, 2014
  • [6] Kai Cordes, Jörn Ostermann: Increasing the Precision of Junction Shaped Features, International Conference on Machine Vision Applications (MVA), IEEE Computer Society, 2015
  • [7] Thao-Ngoc Nguyen and Kazunori Miyata: Multi-scale Region Perpendicular local binary pattern: An Effective Feature for Interest Region Description. The Visual Computer 31(4), 391--406, Springer, 2015
  • [8] Kai Cordes, Lukas Grundmann, and Jörn Ostermann: Feature Evaluation with High-Resolution Images, Computer Analysis of Images and Patterns (CAIP), Springer, 2015
  • [9] Zhao Chunyang and Zhao Huaici: Accurate and robust feature-based homography estimation using HALF-SIFT and feature localization error weighting. Journal of Visual Communication and Image Representation, 40, 2016: 288-299
  • [10]  Vassileios Balntas, Karel Lenc: Local Features: State of the art, open problems and performance evaluation,  Computer Vision – ECCV 2016 Workshops, Springer International Publishing
  • [11] Elena Ranguelova, "A Salient Region Detector for structured images," 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pp. 1-8
  • [12] Kohei Kawai et al. "Compactification of Affine Transformation Filter Using Tensor Decomposition", ICIP 2018, pp.2162-2166

  • Conference Contributions
    • Kai Cordes, Lukas Grundmann, Jörn Ostermann
      Feature Evaluation with High-Resolution Images
      The 16th International Conference on Computer Analysis of Images and Patterns (CAIP), Lecture Notes in Computer Science (LNCS), Springer International Publishing Switzerland, Vol. 9256, pp. 374--386, Valletta, Malta, September 2015, edited by George Azzopardi and Nicolai Petkov
    • Kai Cordes, Bodo Rosenhahn, Jörn Ostermann
      High-Resolution Feature Evaluation Benchmark
      The 15th International Conference on Computer Analysis of Images and Patterns (CAIP), Lecture Notes in Computer Science (LNCS), Springer Berlin/Heidelberg, Vol. 8047, pp. 327-334, York, UK, August 2013, edited by Richard Wilson et al.
    • Kai Cordes, Bodo Rosenhahn, Jörn Ostermann
      Increasing the Accuracy of Feature Evaluation Benchmarks Using Differential Evolution
      IEEE Symposium Series on Computational Intelligence (SSCI) - IEEE Symposium on Differential Evolution (SDE), IEEE Computer Society, Paris, France, April 2011