Übungsbetreuung:

Machine learning (ML) and especially Deep Learning (DL) achieved many breakthroughs in the last years. However, ML/DL applications not only require data, but also a lot of expert knowledge to apply it successfully.

Major challenges for new applications include (a) the choice of the algorithm to be used (e.g., SVM, random forest, deep neural network) and (b) its hyperparameter settings. These design decisions are crucial in order to obtain a model with peak performance. Plus, they have to be made for each new task or dataset at hand. This is particularly hard for deep learning because of the vast possible neural network architectures and the many hyperparameters, e.g., the learning rate for the optimizer. Furthermore, training a neural network takes time: minutes, hours or even weeks, such that it is impossible to manually try all hyperparameter configurations. Overall, these design decisions require a lot of expertise, the design process is time-consuming and the manual tuning is a tedious and error-prone task. Therefore, we need more efficient and automatic approaches.

We will discuss approaches and meta-systems that automate the process of obtaining well-performing machine learning systems, so-called Automated Machine Learning (AutoML). AutoML systems allow for faster development of new ML/DL applications, require far less expert knowledge and often even outperform human developers.

In this seminar we will focus on algorithms and state-of-the-art in the field of AutoML by discussing relevant publications.

We will discuss following topics (not exhaustive):

- Algorithm Selection
- Hyperparameter Optimization (HPO) (Bayesian Optimization, Evolutionary Algorithms, Single-Criterion vs. Multi-Criteria Optimization)
- Speed-Up Techniques for HPO (e.g., Multi-Fidelity Optimization)
- Algorithm Configuration
- Meta-Learning (Learning to Learn, Population-Based Training, Dynamic Algorithm Configuration (DAC))

We strongly recommend that you know the foundations of machine learning in order to attend the course. You should have attended at least one other course for ML in the past.

Preferably you are also familiar at least one of:

- Bayesian Optimization
- Evolutionary Algorithms
- Deep Learning
- Reinforcement Learning

The course will be held in English. We will have weekly sessions in which one or two students will present a paper each (in English). The paper can be any of those from the paper list. After the presentation you will discuss the paper in small groups and afterwards briefly together in the plenary.

So in total, you will be expected to

- attend seminars and read the current paper beforehand
- present a paper yourself
- write a scientific report

Your grade will consist of your presentation and the scientific report. The scientific report is a written discussion of the paper you chose, results of the discussion of the small group and the plenary best to be included.

In the end, you will

- have read and discussed many relevant papers in the field of AutoML
- have improved your scientific presentation and writing skills

The full list of papers can be found on the StudIP course page.