Glossary

SMAC

Sequential Model-Based Algorithm Configuration.

BO

See Bayesian Optimization.

HB

See Hyperband.

BOHB

Bayesian optimization and Hyperband.

ROAR

See Random Online Adaptive Racing.

BB

See Black-Box.

MF

See Multi-Fidelity.

RF

Random Forest.

GP

Gaussian Process.

GP-MCMC

Gaussian Process with Markov-Chain Monte-Carlo.

CV

Cross-Validation.

CLI

Command-Line Interface.

HP

Hyperparameter.

Bayesian Optimization

Bayesian optimization is a sequential design strategy for global optimization of black-box functions that does not assume any functional forms. It is usually employed to optimize expensive-to-evaluate functions. A Bayesian optimization weights exploration and exploitation to find the minimum of its objective.

Hyperband

Hyperband. A novel bandit-based algorithm for hyperparameter optimization. Hyperband is an extension of successive halving and therefore works with multi-fidelities.

Random Online Adaptive Racing

Random Online Adaptive Racing. A simple model-free instantiation of the general SMBO framework. It selects configurations uniformly random and iteratively compares them against the current incumbent using the intensification mechanism. See SMAC extended chapter 3.2 for details.

Black-Box

Refers to an algorithm being optimized, where only input and output are observable.

Target Function

Your model, which returns a cost based on the given config, seed, budget, and/or instance.

Trial

Trial is a single run of a target function on a combination of configuration, seed, budget and/or instance.

Objective

An objective is a metric to evaluate the quality or performance of an algorithm.

Multi-Objective

A multi-objective optimization problem is a problem with more than one objective. The goal is to find a solution that is optimal or at least a good compromise in all objectives.

Budget

Budget is another word for fidelity. Examples are the number of training epochs or the size of the data subset the algorithm is trained on. However, budget can also be used in the context of instances. For example, if you have 100 instances (let’s say we optimize across datasets) and you want to run your algorithm on 10 of them, then the budget is 10.

Multi-Fidelity

Multi-fidelity refers to running an algorithm on multiple budgets (such as number of epochs or subsets of data) and thereby evaluating the performance prematurely.

Instances

Often you want to optimize across different datasets, subsets, or even different transformations (e.g. augmentation). In general, each of these is called an instance. Configurations are evaluated on multiple instances so that a configuration found which performs superior on all instances instead of only a few.

Intensification

A mechanism, that governs how many evaluations to perform with each configuration and when to trust a configuration enough to make it the new current best known configuration (the incumbent).

Incumbent

The incumbent is the current best known configuration.