Conditional Coding for Learned Image and Video Compression

TNT members involved in this project:
Martin Benjak, M. Sc.
Prof. Dr.-Ing. Jörn Ostermann

In this project we investigate learning-based video coding from the point of view of conditional coding with a meta-learning-based method for regularization and dynamic adaptation in cooperation with the Department of Computer Science of the National Chiao Tung University in Taiwan.

The development of new deep neural network architectures such as variational autoencoders (VAE) and augmented normalizing flows (ANF) opens up new possibilities for learning-based video coding. In this project, we develop a new method for conditional video coding based on ANFs instead of the often used VAEs. ANFs offer the advantage that they are more expressive than VAEs, but still include VAEs as a special case. In another aspect of this project, we deal with the adaptability and generalization ability of a learning-based video codec. A weakness of learning-based video codecs is the large deviation in data distribution between training and test data. This means that a codec achieves good results on the training data, but poor results on unknown data. To improve generalization capabilities, we develop a meta cost function that allows preserving common properties between frames of a video sequence. Furthermore, we use this meta cost function to dynamically adapt the decoder to the data distribution of the input data during inference.

Show all publications
  • Yi-Hsin Chen, Chen-Wei Cheng, Martin Benjak, Wen-Hsiao Peng
    Progress report on MaskCRT
    15th Meeting of ISO/IEC JTC 1/SC 29/AG 5 Document m66976, April 2024
  • Yi-Hsin Chen, Cheng-Wei Chen, Zong-Lin Gao, Martin Benjak, Wen-Hsiao Peng
    Results on the Bit Rate Fitting for MaskCRT
    15th Meeting of ISO/IEC JTC 1/SC 29/AG 5 Document m67455, April 2024
  • Yi-Hsin Chen, Zong-Lin Gao, Martin Benjak, Wen-Hsiao Peng
    Response to Call for Learning-Based Video Codecs for Study of Quality Assessment by NYCU and LUH
    14th Meeting of ISO/IEC JTC 1/SC 29/AG 5 Document m66163, January 2024