Usage Recommendation

SMAC of course itself offers a lot of design choices, some of which are crucial to achieve peak performance. Luckily, often it is sufficient to distinguish between a few problem classes. To make the usage of SMAC as easy as possible, we provide several facades designed for these different use cases. Here we give some general recommendations on when to use which facade. These recommendations are based on our experience and technical limitations and is by far not intended to be complete:

SMAC4BO

SMAC4HPO

SMAC4AC

# parameter

low

low/medium/high

low/medium/high

categorical parameter

yes

supported

supported

conditional parameter

yes

supported

supported

instances

no

None or CV-folds

yes

stochasticity

no

supported

supported

objective

any (except runtime)

e.g. validation loss

e.g. runtime or quality

Some examples of typical use cases:

SMAC4BO: Bayesian Optimization using a Gaussian Process and Expected Improvement
  • Optimizing the objective value of Branin and other low dimensional artificial test functions

  • Finding the best learning rate for training a neural network wrt. RMSE on a validation dataset

  • Optimizing the choice of kernel and penalty of a SVM wrt. RMSE on a validation dataset

SMAC4HPO: Bayesian optimization using a Random Forest
  • Finding the optimal choice of machine learning algorithm and its hyperparameters wrt. validation error

  • Tuning the architecture and training parameters of a neural network wrt. classification error on a validation dataset

  • Optimize hyperparameters of a SVM wrt. the CV-fold error

  • Minimize objective values of problems that are noisy and/or yield crashed runs (e.g. due to mem-outs).

  • Finding the best setting of an RL-agent to minimize regret (or a set of RL problems)

SMAC4AC: Algorithm configuration using a Random Forest
  • Minimizing the average time it takes for a SAT-solver to solve a set of SAT instances

  • Configuring a MIP solver to solve a set of mixed-integer-problems as fast as possible

  • Optimizing the average quality of solutions returned by a configurable TSP solver

Important: If your problem is not covered in this table, this doesn’t mean you can’t benefit from using our tool. In case of doubt, please create an issue on Github.