Automated Machine Learning

TNT members involved in this project:
Carolin Benjamins, M.Sc.
Difan Deng, M. Sc.
MSc. Theresa Eimer
Prof. Dr. rer. nat. Marius Lindauer
René Sass

To use machine learning (ML), users have to choose between many design options: (i) ML algorithms (ii) pre-processing techniques, (iii) post-processing techniques, (iv) hyperparameter settings, (v) architectures of neural networks and so on. These design decisions are often responsible whether ML systems return random predictions or achieve state-of-the-art performance. Unfortunately, even for ML-experts it is a tedious and error-prone task and thus it is not easy to make these decisions efficiently.

Automated machine learning (AutoML) addresses this challenge by automating the design process such that AutoML tools support users to efficiently develop new ML applications.

Hyperparameter Optimization and Bayesian Optimization

To achieve peak-performance with an algorithm, choosing an appropriate hyperparameter configuration is crucial. Since hyperparameters are often not very intuitive for human developers, it is a tedious and error-prone task to choose these settings. Bayesian Optimization is a sample-efficient approach to find such hyperparameter configurations in an automatic way, saving human developers tremendous amounts of development time.

Neural Architecture Search

Applying deep learning to new datasets also requires to find a well-performing architecture of a deep neural network. Such an architecture influences the performance, but also other metrics, such as inference time, memory consumption etc pp. Unfortunately, it is again not obvious for human developers how to design such deep neural networks making the process fairly inefficient. Neural architecture search is an paradigma to automatically determine the best architectures for new datasets, making new applications of deep learning feasible also at larger scale.

Dynamic Algorithm Configuration

Instead of choosing the hyperparameters of an ML algorithm once, many hyperparameters have to be adapted over time. A well-known example is the learning rate of a deep neural network, which is decreased, sometimes also increased, over time. So far, these dynamic hyperparameters are controlled by a human-designed heuristic, which is often not optimal for a new dataset. Therefore, we develop new approaches for dynamic algorithm configuration, which learns from data how to adjust these on-the-fly.

Interpretability of AutoML 

A major drawback of AutoML tools is the risk that ML will be even a more mysterious black box than it ever was. Therefore, we also develop analysis tools that provide feedback to AutoML users about important insights, such as, (i) how to use AutoML tools more efficiently or (ii) which hyperparameter decisions were important to achieve the final performance. This helps ML developers to get a better understanding of why and how ML and AutoML works.

Show all publications
  • Artur Souza, Luigi Nardi, Leonardo Oliveira, Kunle Olukotun, Marius Lindauer, Frank Hutter
    Bayesian Optimization with a Prior for the Optimum
    Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), September 2021
  • Lucas Zimmer, Marius Lindauer, Frank Hutter
    Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
    IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE, Vol. 43, No. 9, pp. 3079 - 3090, August 2021
  • Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, Isabelle Gyon, Sirui Hong, Frank Hutter, Rongrong Ji, Julio Jacques Junior, Ge Li, Marius Lindauer, Zhipeng Luo, Meysam Madadi, Thomas Nierhoff, Kangning Niu, Chunguang Pan, Danny Stoll, Sebastien Treguer, Wang Jin, Peng Wang, Chenglin Wu, Xiong Youcheng, Arber Zela, Yang Zhang
    Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019
    IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE, Vol. 43, No. 9, pp. 3108 - 3125, August 2021