smac.facade

Interfaces

class smac.facade.AbstractFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: object

Facade is an abstraction on top of the SMBO backend to organize the components of a Bayesian Optimization loop in a configurable and separable manner to suit the various needs of different (hyperparameter) optimization pipelines.

With the exception to scenario and target_function, which are expected of the user, the parameters model, acquisition_function, acquisition_maximizer, initial_design, random_design, intensifier, multi_objective_algorithm, runhistory_encoder can either be explicitly specified in the subclasses’ get_* methods (defining a specific BO pipeline) or be instantiated by the user to overwrite a pipeline components explicitly.

Parameters:

  • scenario (Scenario) – The scenario object, holding all environmental information.

  • target_function (Callable | str | AbstractRunner) – This function is called internally to judge a trial’s performance. If a string is passed, it is assumed to be a script. In this case, TargetFunctionScriptRunner is used to run the script.

  • model (AbstractModel | None, defaults to None) – The surrogate model.

  • acquisition_function (AbstractAcquisitionFunction | None, defaults to None) – The acquisition function.

  • acquisition_maximizer (AbstractAcquisitionMaximizer | None, defaults to None) – The acquisition maximizer, deciding which configuration is most promising based on the surrogate model and acquisition function.

  • initial_design (InitialDesign | None, defaults to None) – The sampled configurations from the initial design are evaluated before the Bayesian optimization loop starts.

  • random_design (RandomDesign | None, defaults to None) – The random design is used in the acquisition maximizer, deciding whether the next configuration should be drawn from the acquisition function or randomly.

  • intensifier (AbstractIntensifier | None, defaults to None) – The intensifier decides which trial (combination of configuration, seed, budget and instance) should be run next.

  • multi_objective_algorithm (AbstractMultiObjectiveAlgorithm | None, defaults to None) – In case of multiple objectives, the objectives need to be interpreted so that an optimization is possible. The multi-objective algorithm takes care of that.

  • runhistory_encoder (RunHistoryEncoder | None, defaults to None) – Based on the runhistory, the surrogate model is trained. However, the data first needs to be encoded, which is done by the runhistory encoder. For example, inactive hyperparameters need to be encoded or cost values can be log transformed.

  • logging_level (int | Path | None) – The level of logging (the lowest level 0 indicates the debug level). If a path is passed, a yaml file is expected with the logging configuration. If nothing is passed, the default logging.yml from SMAC is used.

  • callbacks (list[Callback], defaults to []) – Callbacks, which are incorporated into the optimization loop.

  • overwrite (bool, defaults to False) – When True, overwrites the run results if a previous run is found that is inconsistent in the meta data with the current setup. If overwrite is set to False, the user is asked for the exact behaviour (overwrite completely, save old run, or use old results).

ask()[source]

Asks the intensifier for the next trial.

Return type:

TrialInfo

abstract static get_acquisition_function(scenario)[source]

Returns the acquisition function instance used in the BO loop, defining the exploration/exploitation trade-off.

Return type:

AbstractAcquisitionFunction

abstract static get_acquisition_maximizer(scenario)[source]

Returns the acquisition optimizer instance to be used in the BO loop, specifying how the acquisition function instance is optimized.

Return type:

AbstractAcquisitionMaximizer

static get_config_selector(scenario, *, retrain_after=8, retries=16)[source]

Returns the default configuration selector.

Return type:

ConfigSelector

abstract static get_initial_design(scenario)[source]

Returns an instance of the initial design class to be used in the BO loop, specifying how the configurations the BO loop is ‘warm-started’ with are selected.

Return type:

AbstractInitialDesign

abstract static get_intensifier(scenario)[source]

Returns the intensifier instance to be used in the BO loop, specifying how to challenge the incumbent configuration on other problem instances.

Return type:

AbstractIntensifier

abstract static get_model(scenario)[source]

Returns the surrogate cost model instance used in the BO loop.

Return type:

AbstractModel

abstract static get_multi_objective_algorithm(scenario)[source]

Returns the multi-objective algorithm instance to be used in the BO loop, specifying the scalarization strategy for multiple objectives’ costs.

Return type:

AbstractMultiObjectiveAlgorithm

abstract static get_random_design(scenario)[source]

Returns an instance of the random design class to be used in the BO loop, specifying how to interleave the BO iterations with randomly selected configurations.

Return type:

AbstractRandomDesign

abstract static get_runhistory_encoder(scenario)[source]

Returns an instance of the runhistory encoder class to be used in the BO loop, specifying how the runhistory is to be prepared for the next surrogate model.

Return type:

AbstractRunHistoryEncoder

property intensifier: AbstractIntensifier

The optimizer which is responsible for the BO loop. Keeps track of useful information like status.

property meta: dict[str, Any]

Generates a hash based on all components of the facade. This is used for the run name or to determine whether a run should be continued or not.

optimize()[source]

Optimizes the configuration of the algorithm.

Returns:

incumbent – Best found configuration.

Return type:

Configuration

property optimizer: SMBO

The optimizer which is responsible for the BO loop. Keeps track of useful information like status.

property runhistory: RunHistory

The runhistory which is filled with all trials during the optimization process.

property scenario: Scenario

The scenario object which holds all environment information.

tell(info, value, save=True)[source]

Adds the result of a trial to the runhistory and updates the intensifier.

Parameters:

  • info (TrialInfo) – Describes the trial from which to process the results.

  • value (TrialValue) – Contains relevant information regarding the execution of a trial.

  • save (bool, optional to True) – Whether the runhistory should be saved.

Return type:

None

validate(config, *, seed=None)[source]

Validates a configuration on seeds different from the ones used in the optimization process and on the highest budget (if budget type is real-valued).

Parameters:

  • config (Configuration) – Configuration to validate

  • instances (list[str] | None, defaults to None) – Which instances to validate. If None, all instances specified in the scenario are used. In case that the budget type is real-valued, this argument is ignored.

  • seed (int | None, defaults to None) – If None, the seed from the scenario is used.

Returns:

cost – The averaged cost of the configuration. In case of multi-fidelity, the cost of each objective is averaged.

Return type:

float | list[float]

class smac.facade.AlgorithmConfigurationFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: AbstractFacade

static get_acquisition_function(scenario, *, xi=0.0)[source]

Returns an Expected Improvement acquisition function.

Parameters:

  • scenario (Scenario) –

  • xi (float, defaults to 0.0) – Controls the balance between exploration and exploitation of the acquisition function.

Return type:

EI

static get_acquisition_maximizer(scenario)[source]

Returns local and sorted random search as acquisition maximizer.

Return type:

LocalAndSortedRandomSearch

static get_initial_design(scenario, *, additional_configs=[])[source]

Returns an initial design, which returns the default configuration.

Parameters:

additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.

Return type:

DefaultInitialDesign

static get_intensifier(scenario, *, max_config_calls=2000, max_incumbents=10)[source]

Returns Intensifier as intensifier. Supports budgets.

Parameters:

  • max_config_calls (int, defaults to 3) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be evaluated at maximum for a configuration.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

Return type:

Intensifier

static get_model(scenario, *, n_trees=10, ratio_features=0.8333333333333334, min_samples_split=3, min_samples_leaf=3, max_depth=20, bootstrapping=True, pca_components=4)[source]

Returns a random forest as surrogate model.

Parameters:

  • n_trees (int, defaults to 10) – The number of trees in the random forest.

  • ratio_features (float, defaults to 5.0 / 6.0) – The ratio of features that are considered for splitting.

  • min_samples_split (int, defaults to 3) – The minimum number of data points to perform a split.

  • min_samples_leaf (int, defaults to 3) – The minimum number of data points in a leaf.

  • max_depth (int, defaults to 20) – The maximum depth of a single tree.

  • bootstrapping (bool, defaults to True) – Enables bootstrapping.

  • pca_components (float, defaults to 4) – Number of components to keep when using PCA to reduce dimensionality of instance features.

Return type:

RandomForest

static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]

Returns the mean aggregation strategy for the multi objective algorithm.

Parameters:

  • scenario (Scenario) –

  • objective_weights (list[float] | None, defaults to None) – Weights for averaging the objectives in a weighted manner. Must be of the same length as the number of objectives.

Return type:

MeanAggregationStrategy

static get_random_design(scenario, *, probability=0.5)[source]

Returns ProbabilityRandomDesign for interleaving configurations.

Parameters:

probability (float, defaults to 0.5) – Probability that a configuration will be drawn at random.

Return type:

ProbabilityRandomDesign

static get_runhistory_encoder(scenario)[source]

Returns the default runhistory encoder.

Return type:

RunHistoryEncoder

class smac.facade.BlackBoxFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: AbstractFacade

static get_acquisition_function(scenario, *, xi=0.0)[source]

Returns an Expected Improvement acquisition function.

Parameters:

  • scenario (Scenario) –

  • xi (float, defaults to 0.0) – Controls the balance between exploration and exploitation of the acquisition function.

Return type:

EI

static get_acquisition_maximizer(scenario, *, challengers=1000, local_search_iterations=10)[source]

Returns local and sorted random search as acquisition maximizer.

Parameters:

  • challengers (int, defaults to 1000) – Number of challengers.

  • local_search_iterations (int, defaults to 10) – Number of local search iterations.

Return type:

LocalAndSortedRandomSearch

static get_initial_design(scenario, *, n_configs=None, n_configs_per_hyperparamter=8, max_ratio=0.25, additional_configs=[])[source]

Returns a Sobol design instance.

Parameters:

  • scenario (Scenario) –

  • n_configs (int | None, defaults to None) – Number of initial configurations (disables the arguments n_configs_per_hyperparameter).

  • n_configs_per_hyperparameter (int, defaults to 8) – Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and n_configs_per_hyperparameter is set to 10, then 50 initial configurations will be samples.

  • max_ratio (float, defaults to 0.25) – Use at most scenario.n_trials * max_ratio number of configurations in the initial design. Additional configurations are not affected by this parameter.

  • additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.

Return type:

SobolInitialDesign

static get_intensifier(scenario, *, max_config_calls=3, max_incumbents=20)[source]

Returns Intensifier as intensifier. Uses the default configuration for race_against.

Parameters:

  • scenario (Scenario) –

  • max_config_calls (int, defaults to 3) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be evaluated at maximum for a configuration.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

Return type:

Intensifier

static get_kernel(scenario)[source]

Returns a kernel for the Gaussian Process surrogate model.

The kernel is a composite of kernels depending on the type of hyperparameters: categorical (HammingKernel), continuous (Matern), and noise kernels (White).

Return type:

Kernel

static get_model(scenario, *, model_type=None, kernel=None)[source]

Returns a Gaussian Process surrogate model.

Parameters:

  • scenario (Scenario) –

  • model_type (str | None, defaults to None) – Which Gaussian Process model should be chosen. Choose between vanilla and mcmc.

  • kernel (kernels.Kernel | None, defaults to None) – The kernel used in the surrogate model.

Returns:

model – The instantiated gaussian process.

Return type:

GaussianProcess | MCMCGaussianProcess

static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]

Returns the mean aggregation strategy for the multi-objective algorithm.

Parameters:

  • scenario (Scenario) –

  • objective_weights (list[float] | None, defaults to None) – Weights for averaging the objectives in a weighted manner. Must be of the same length as the number of objectives.

Return type:

MeanAggregationStrategy

static get_random_design(scenario, *, probability=0.08447232371720552)[source]

Returns ProbabilityRandomDesign for interleaving configurations.

Parameters:

probability (float, defaults to 0.08447232371720552) – Probability that a configuration will be drawn at random.

Return type:

ProbabilityRandomDesign

static get_runhistory_encoder(scenario)[source]

Returns the default runhistory encoder.

Return type:

RunHistoryEncoder

class smac.facade.HyperbandFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: RandomFacade

Facade to use model-free Hyperband [LJDR18] for algorithm configuration.

Uses Random Aggressive Online Racing (ROAR) to compare configurations, a random initial design and the Hyperband intensifier.

static get_intensifier(scenario, *, eta=3, n_seeds=1, instance_seed_order='shuffle_once', max_incumbents=10, incumbent_selection='highest_observed_budget')[source]

Returns a Hyperband intensifier instance. Budgets are supported.

Return type:

Hyperband

etaint, defaults to 3

Input that controls the proportion of configurations discarded in each round of Successive Halving.

n_seedsint, defaults to 1

How many seeds to use for each instance.

instance_seed_orderstr, defaults to “shuffle_once”

How to order the instance-seed pairs. Can be set to: * None: No shuffling at all and use the instance-seed order provided by the user. * “shuffle_once”: Shuffle the instance-seed keys once and use the same order across all runs. * “shuffle”: Shuffle the instance-seed keys for each bracket individually.

incumbent_selectionstr, defaults to “any_budget”

How to select the incumbent when using budgets. Can be set to: * “any_budget”: Incumbent is the best on any budget i.e., best performance regardless of budget. * “highest_observed_budget”: Incumbent is the best in the highest budget run so far. * “highest_budget”: Incumbent is selected only based on the highest budget.

max_incumbentsint, defaults to 10

How many incumbents to keep track of in the case of multi-objective.

class smac.facade.HyperparameterOptimizationFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: AbstractFacade

static get_acquisition_function(scenario, *, xi=0.0)[source]

Returns an Expected Improvement acquisition function.

Parameters:

  • scenario (Scenario) –

  • xi (float, defaults to 0.0) – Controls the balance between exploration and exploitation of the acquisition function.

Return type:

EI

static get_acquisition_maximizer(scenario, *, challengers=10000, local_search_iterations=10)[source]

Returns local and sorted random search as acquisition maximizer.

Warning

If you experience RAM issues, try to reduce the number of challengers.

Parameters:

  • challengers (int, defaults to 10000) – Number of challengers.

  • local_search_iterations (int, defaults to 10) – Number of local search iterations.

Return type:

LocalAndSortedRandomSearch

static get_initial_design(scenario, *, n_configs=None, n_configs_per_hyperparamter=10, max_ratio=0.25, additional_configs=[])[source]

Returns a Sobol design instance.

Parameters:

  • scenario (Scenario) –

  • n_configs (int | None, defaults to None) – Number of initial configurations (disables the arguments n_configs_per_hyperparameter).

  • n_configs_per_hyperparameter (int, defaults to 10) – Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and n_configs_per_hyperparameter is set to 10, then 50 initial configurations will be samples.

  • max_ratio (float, defaults to 0.25) – Use at most scenario.n_trials * max_ratio number of configurations in the initial design. Additional configurations are not affected by this parameter.

  • additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.

Return type:

SobolInitialDesign

static get_intensifier(scenario, *, max_config_calls=3, max_incumbents=10)[source]

Returns Intensifier as intensifier. Uses the default configuration for race_against.

Parameters:

  • scenario (Scenario) –

  • max_config_calls (int, defaults to 3) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be max evaluated for a configuration.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

Return type:

Intensifier

static get_model(scenario, *, n_trees=10, ratio_features=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=1048576, bootstrapping=True)[source]

Returns a random forest as surrogate model.

Parameters:

  • n_trees (int, defaults to 10) – The number of trees in the random forest.

  • ratio_features (float, defaults to 5.0 / 6.0) – The ratio of features that are considered for splitting.

  • min_samples_split (int, defaults to 3) – The minimum number of data points to perform a split.

  • min_samples_leaf (int, defaults to 3) – The minimum number of data points in a leaf.

  • max_depth (int, defaults to 20) – The maximum depth of a single tree.

  • bootstrapping (bool, defaults to True) – Enables bootstrapping.

Return type:

RandomForest

static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]

Returns the mean aggregation strategy for the multi-objective algorithm.

Parameters:

  • scenario (Scenario) –

  • objective_weights (list[float] | None, defaults to None) – Weights for averaging the objectives in a weighted manner. Must be of the same length as the number of objectives.

Return type:

MeanAggregationStrategy

static get_random_design(scenario, *, probability=0.2)[source]

Returns ProbabilityRandomDesign for interleaving configurations.

Parameters:

probability (float, defaults to 0.2) – Probability that a configuration will be drawn at random.

Return type:

ProbabilityRandomDesign

static get_runhistory_encoder(scenario)[source]

Returns a log scaled runhistory encoder. That means that costs are log scaled before training the surrogate model.

Return type:

RunHistoryLogScaledEncoder

class smac.facade.MultiFidelityFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: HyperparameterOptimizationFacade

This facade configures SMAC in a multi-fidelity setting.

static get_initial_design(scenario, *, n_configs=None, n_configs_per_hyperparamter=10, max_ratio=0.25, additional_configs=[])[source]

Returns a random initial design.

Parameters:

  • scenario (Scenario) –

  • n_configs (int | None, defaults to None) – Number of initial configurations (disables the arguments n_configs_per_hyperparameter).

  • n_configs_per_hyperparameter (int, defaults to 10) – Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and n_configs_per_hyperparameter is set to 10, then 50 initial configurations will be samples.

  • max_ratio (float, defaults to 0.1) – Use at most scenario.n_trials * max_ratio number of configurations in the initial design. Additional configurations are not affected by this parameter.

  • additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.

Return type:

RandomInitialDesign

static get_intensifier(scenario, *, eta=3, n_seeds=1, instance_seed_order='shuffle_once', max_incumbents=10, incumbent_selection='highest_observed_budget')[source]

Returns a Hyperband intensifier instance. Budgets are supported.

Return type:

Hyperband

etaint, defaults to 3

Input that controls the proportion of configurations discarded in each round of Successive Halving.

n_seedsint, defaults to 1

How many seeds to use for each instance.

instance_seed_orderstr, defaults to “shuffle_once”

How to order the instance-seed pairs. Can be set to: * None: No shuffling at all and use the instance-seed order provided by the user. * “shuffle_once”: Shuffle the instance-seed keys once and use the same order across all runs. * “shuffle”: Shuffles the instance-seed keys for each bracket individually.

incumbent_selectionstr, defaults to “any_budget”

How to select the incumbent when using budgets. Can be set to: * “any_budget”: Incumbent is the best on any budget, i.e., the best performance regardless of budget. * “highest_observed_budget”: Incumbent is the best in the highest budget run so far. * “highest_budget”: Incumbent is selected only based on the highest budget.

max_incumbentsint, defaults to 10

How many incumbents to keep track of in the case of multi-objective.

class smac.facade.RandomFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, config_selector=None, logging_level=None, callbacks=[], overwrite=False)[source]

Bases: AbstractFacade

Facade to use Random Online Aggressive Racing (ROAR).

Aggressive Racing: When we have a new configuration θ, we want to compare it to the current best configuration, the incumbent θ*. ROAR uses the ‘racing’ approach, where we run few times for unpromising θ and many times for promising configurations. Once we are confident enough that θ is better than θ*, we update the incumbent θ* ⟵ θ. Aggressive means rejecting low-performing configurations very early, often after a single run. This together is called aggressive racing.

ROAR Loop: The main ROAR loop looks as follows:

  1. Select a configuration θ uniformly at random.

  2. Compare θ to incumbent θ* online (one θ at a time):

  • Reject/accept θ with aggressive racing

Setup: Uses a random model and random search for the optimization of the acquisition function.

Note

The surrogate model and the acquisition function is not used during the optimization and therefore replaced by dummies.

static get_acquisition_function(scenario)[source]

The random facade is not using an acquisition function. Therefore, we simply return a dummy function.

Return type:

AbstractAcquisitionFunction

static get_acquisition_maximizer(scenario)[source]

We return RandomSearch as maximizer which samples configurations randomly from the configuration space and therefore neither uses the acquisition function nor the model.

Return type:

RandomSearch

static get_initial_design(scenario, *, additional_configs=[])[source]

Returns an initial design, which returns the default configuration.

Parameters:

additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.

Return type:

DefaultInitialDesign

static get_intensifier(scenario, *, max_config_calls=3, max_incumbents=10)[source]

Returns Intensifier as intensifier.

Note

Please use the HyperbandFacade if you want to incorporate budgets.

Warning

If you are in an algorithm configuration setting, consider increasing max_config_calls.

Parameters:

  • max_config_calls (int, defaults to 3) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be max evaluated for a configuration.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

Return type:

Intensifier

static get_model(scenario)[source]

The model is used in the acquisition function. Since we do not use an acquisition function, we return a dummy model (returning random values in this case).

Return type:

RandomModel

static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]

Returns the mean aggregation strategy for the multi-objective algorithm.

Parameters:

  • scenario (Scenario) –

  • objective_weights (list[float] | None, defaults to None) – Weights for averaging the objectives in a weighted manner. Must be of the same length as the number of objectives.

Return type:

MeanAggregationStrategy

static get_random_design(scenario)[source]

Just like the acquisition function, we do not use a random design. Therefore, we return a dummy design.

Return type:

AbstractRandomDesign

static get_runhistory_encoder(scenario)[source]

Returns the default runhistory encoder.

Return type:

RunHistoryEncoder

Modules

smac.facade.abstract_facade

smac.facade.algorithm_configuration_facade

smac.facade.blackbox_facade

smac.facade.hyperband_facade

smac.facade.hyperparameter_optimization_facade

smac.facade.multi_fidelity_facade

smac.facade.random_facade

previous

API References

next

smac.facade.abstract_facade