Glossary

BO

Bayesian Optimization. A Black-Box optimization algorithm weighing exploration & exploitation to find the minimum of its objective.

SMAC

Sequential Model-Based Algorithm Configuration.

ROAR

Random Online Adaptive Racing. A simple model-free instantiation of the general SMBO framework. It selects configurations uniformly random and iteratively compares them against the current incumbent using the intensification mechanism. See SMAC extended chapter 3.2 for details.

BB

Black-Box. Refers to an algorithm being optimized, where only input & output are observable.

MF

Multi-Fidelity. Refers to running an algorithm on multiple budgets (such as number of epochs or subsets of data) and thereby evaluating the performance prematurely.

TAE

Target Algorithm Evaluator. Your model, which returns a cost based on the given config, budget and instance.

RF

Random Forest.

GP

Gaussian Process.

GP-MCMC

Gaussian Process with Markov-Chain Monte-Carlo.

Objective

An objective is a metric to evaluate the quality or performance of an Algorithm.

Budget

Budget is another word for fidelity. Examples are the number of training epochs or the size of the data subset the algorithm is trained on.

PCS

ConfigurationSpace can be written/read from a PCS file.

EPM

Empirical Performance Models. Empirical performance models are regression models that characterize a given algorithm’s performance across problem instances and/or parameter settings. These models can predict the performance of algorithms on previously unseen input, including previously unseen problem instances and or previously untested parameter settings and are useful for analyzing of how an algorithm performs under different conditions, select promising configurations for a new problem instance, or surrogate benchmarks.

Intensification

A mechanism, that governs how many evaluations to perform with each configuration and when to trust a configuration enough to make it the new current best known configuration (the incumbent).

CV

Cross-Validation.

CLI

Command-Line Interface.