smac.facade¶
Interfaces¶
- class smac.facade.AbstractFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
object
Facade is an abstraction on top of the SMBO backend to organize the components of a Bayesian Optimization loop in a configurable and separable manner to suit the various needs of different (hyperparameter) optimization pipelines.
With the exception to scenario and
target_function
, which are expected of the user, the parametersmodel
,acquisition_function
,acquisition_maximizer
,initial_design
,random_design
,intensifier
,multi_objective_algorithm
,runhistory_encoder
can either be explicitly specified in the subclasses’get_*
methods (defining a specific BO pipeline) or be instantiated by the user to overwrite a pipelines components explicitly.- Parameters:
scenario (Scenario) – The scenario object, holding all environmental information.
target_function (Callable | str | AbstractRunner) – This function is called internally to judge a trial’s performance. If a string is passed, it is assumed to be a script. In this case,
TargetFunctionScriptRunner
is used to run the script.model (AbstractModel | None, defaults to None) – The surrogate model.
acquisition_function (AbstractAcquisitionFunction | None, defaults to None) – The acquisition function.
acquisition_maximizer (AbstractAcquisitionMaximizer | None, defaults to None) – The acquisition maximizer, deciding which configuration is most promising based on the surrogate model and acquisition function.
initial_design (InitialDesign | None, defaults to None) – The sampled configurations from the initial design are evaluated before the Bayesian optimization loop starts.
random_design (RandomDesign | None, defaults to None) – The random design is used in the acquisition maximier, deciding whether the next configuration should be drawn from the acquisition function or randomly.
intensifier (AbstractIntensifier | None, defaults to None) – The intensifier decides which trial (combination of configuration, seed, budget and instance) should be run next.
multi_objective_algorithm (AbstractMultiObjectiveAlgorithm | None, defaults to None) – In case of multiple objectives, the objectives need to be interpreted so that an optimization is possible. The multi objective algorithm takes care of that.
runhistory_encoder (RunHistoryEncoder | None, defaults to None) – Based on the runhistory, the surrogate model is trained. However, the data first needs to be encoded, which is done by the runhistory encoder. For example, inactive hyperparameters need to be encoded or cost values can be log transformed.
logging_level (int | Path | None) – The level of logging (the lowest level 0 indicates the debug level). If a path is passed, a yaml file is expected with the logging configuration. If nothing is passed, the default logging.yml from SMAC is used.
callbacks (list[Callback], defaults to []) – Callbacks, which are incorporated into the optimization loop.
overwrite (bool, defaults to False) – When True, overwrites the run results if a previous run is found that is inconsistent in the meta data with the current setup. If
overwrite
is set to False, the user is asked for the exact behaviour (overwrite completely, save old run, or use old results).
- ask()[source]¶
Asks the intensifier for the next trial. This method returns only trials with the intend to run.
- Return type:
- abstract static get_acquisition_function(scenario)[source]¶
Returns the acquisition function instance used in the BO loop, defining the exploration/exploitation trade-off.
- Return type:
- abstract static get_acquisition_maximizer(scenario)[source]¶
Returns the acquisition optimizer instance to be used in the BO loop, specifying how the acquisition function instance is optimized.
- Return type:
- abstract static get_initial_design(scenario)[source]¶
Returns an instance of the initial design class to be used in the BO loop, specifying how the configurations the BO loop is ‘warm-started’ with are selected.
- Return type:
- abstract static get_intensifier(scenario)[source]¶
Returns the intensifier instance to be used in the BO loop, specifying how to challenge the incumbent configuration on other problem instances.
- Return type:
- abstract static get_model(scenario)[source]¶
Returns the surrogate cost model instance used in the BO loop.
- Return type:
- abstract static get_multi_objective_algorithm(scenario)[source]¶
Returns the multi-objective algorithm instance to be used in the Bayesian optimization loop, specifying the scalarization strategy for multiple objectives’ costs
- Return type:
- get_next_configurations()[source]¶
Choose next candidate solution with Bayesian optimization. The suggested configurations depend on the surrogate model acquisition optimizer/function. This method is used by the intensifier.
- Return type:
Iterator
[Configuration
]
- abstract static get_random_design(scenario)[source]¶
Returns an instance of the random design class to be used in the BO loop, specifying how to interleave the BO iterations with randomly selected configurations.
- Return type:
- abstract static get_runhistory_encoder(scenario)[source]¶
Returns an instance of the runhistory encoder class to be used in the Bayesian optimization loop, specifying how the runhistory is to be prepared for the next surrogate model.
- Return type:
- get_target_function_budgets()[source]¶
Which budgets are used to call the target function.
- Return type:
list
[Optional
[float
]]
- get_target_function_instances()[source]¶
Which instances are used to call the target function.
- Return type:
list
[Optional
[str
]]
- get_target_function_seeds()[source]¶
Which seeds are used to call the target function.
- Return type:
list
[int
]
- property incumbent: ConfigSpace.configuration_space.Configuration | None¶
The best configuration so far.
- Return type:
Optional
[Configuration
]
- property meta: dict[str, Any]¶
Generates a hash based on all components of the facade. This is used for the run name or to determine whether a run should be continued or not.
- Return type:
dict
[str
,Any
]
- optimize()[source]¶
Optimizes the algorithm.
- Returns:
incumbent – Best found configuration.
- Return type:
Configuration
- property runhistory: RunHistory¶
The run history, which is filled with all information during the optimization process.
- Return type:
- property stats: Stats¶
The stats object, which is updated during the optimization and shows relevant information, e.g., how many trials have been finished and how the trajectory looks like.
- Return type:
Stats
- tell(info, value, save=True)[source]¶
Adds the result of a trial to the runhistory and updates the intensifier. Also, the stats object is updated.
- Parameters:
info (TrialInfo) – Describes the trial from which to process the results.
value (TrialValue) – Contains relevant information regarding the execution of a trial.
save (bool, optional to True) – Whether the runhistory should be saved.
- Return type:
None
- validate(config, *, instances=None, seed=None)[source]¶
Validates a configuration with different seeds than in the optimization process and on the highest budget (if budget type is real-valued).
- Parameters:
config (Configuration) – Configuration to validate
instances (list[str] | None, defaults to None) – Which instances to validate. If None, all instances specified in the scenario are used. In case that the budget type is real-valued budget, this argument is ignored.
seed (int | None, defaults to None) – If None, the seed from the scenario is used.
- Returns:
cost – The averaged cost of the configuration. In case of multi-fidelity, the cost of each objective is averaged.
- Return type:
float | list[float]
- class smac.facade.AlgorithmConfigurationFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
AbstractFacade
- static get_acquisition_function(scenario, *, xi=0.0)[source]¶
Returns an Expected Improvement acquisition function.
- static get_acquisition_maximizer(scenario)[source]¶
Returns local and sorted random search as acquisition maximizer.
- Return type:
- static get_initial_design(scenario, *, additional_configs=[])[source]¶
Returns an initial design, which returns the default configuration.
- Parameters:
additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.
- Return type:
- static get_intensifier(scenario, *, min_config_calls=1, max_config_calls=2000, min_challenger=1, intensify_percentage=0.5)[source]¶
Returns
Intensifier
as intensifier. Uses the default configuration forrace_against
.- Parameters:
scenario (Scenario) –
min_config_calls (int, defaults to 1) – Minimum number of trials per config (summed over all calls to intensify).
max_config_calls (int, defaults to 2000) – Maximum number of trials per config (summed over all calls to intensify).
min_challenger (int, defaults to 2) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).
intensify_percentage (float, defaults to 0.5) – How much percentage of the time should configurations be intensified (evaluated on higher budgets or more instances). This parameter is accessed in the SMBO class.
- Return type:
- static get_model(scenario, *, n_trees=10, ratio_features=0.8333333333333334, min_samples_split=3, min_samples_leaf=3, max_depth=20, bootstrapping=True, pca_components=4)[source]¶
Returns a random forest as surrogate model.
- Parameters:
n_trees (int, defaults to 10) – The number of trees in the random forest.
ratio_features (float, defaults to 5.0 / 6.0) – The ratio of features that are considered for splitting.
min_samples_split (int, defaults to 3) – The minimum number of data points to perform a split.
min_samples_leaf (int, defaults to 3) – The minimum number of data points in a leaf.
max_depth (int, defaults to 20) – The maximum depth of a single tree.
bootstrapping (bool, defaults to True) – Enables bootstrapping.
pca_components (float, defaults to 4) – Number of components to keep when using PCA to reduce dimensionality of instance features.
- Return type:
- static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]¶
Returns the mean aggregation strategy for the multi objective algorithm.
- Parameters:
scenario (Scenario) –
objective_weights (list[float] | None, defaults to None) – Weights for an weighted average. Must be of the same length as the number of objectives.
- Return type:
- static get_random_design(scenario, *, probability=0.5)[source]¶
Returns
ProbabilityRandomDesign
for interleaving configurations.- Parameters:
probability (float, defaults to 0.5) – Probability that a configuration will be drawn at random.
- Return type:
- class smac.facade.BlackBoxFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
AbstractFacade
- static get_acquisition_function(scenario, *, xi=0.0)[source]¶
Returns an Expected Improvement acquisition function.
- static get_acquisition_maximizer(scenario, *, challengers=1000, local_search_iterations=10)[source]¶
Returns local and sorted random search as acquisition maximizer.
- Parameters:
challengers (int, defaults to 1000) – Number of challengers.
local_search_iterations (int, defauts to 10) – Number of local search iterations.
- Return type:
- static get_initial_design(scenario, *, n_configs=None, n_configs_per_hyperparamter=10, max_ratio=0.1, additional_configs=[])[source]¶
Returns a Sobol design instance.
- Parameters:
scenario (Scenario) –
n_configs (int | None, defaults to None) – Number of initial configurations (disables the arguments
n_configs_per_hyperparameter
).n_configs_per_hyperparameter (int, defaults to 10) – Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and
n_configs_per_hyperparameter
is set to 10, then 50 initial configurations will be samples.max_ratio (float, defaults to 0.1) – Use at most
scenario.n_trials
*max_ratio
number of configurations in the initial design. Additional configurations are not affected by this parameter.additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.
- Return type:
- static get_intensifier(scenario, *, min_challenger=1, min_config_calls=1, max_config_calls=3, intensify_percentage=0.5)[source]¶
Returns
Intensifier
as intensifier. Uses the default configuration forrace_against
.- Parameters:
scenario (Scenario) –
min_config_calls (int, defaults to 1) – Minimum number of trials per config (summed over all calls to intensify).
max_config_calls (int, defaults to 1) – Maximum number of trials per config (summed over all calls to intensify).
min_challenger (int, defaults to 3) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).
intensify_percentage (float, defaults to 0.5) – How much percentage of the time should configurations be intensified (evaluated on higher budgets or more instances). This parameter is accessed in the SMBO class.
- Return type:
- static get_kernel(scenario)[source]¶
Returns a kernel for the Gaussian Process surrogate model.
The kernel is a composite of kernels depending on the type of hyperparameters: categorical (HammingKernel), continuous (Matern), and noise kernels (White).
- Return type:
Kernel
- static get_model(scenario, *, model_type=None, kernel=None)[source]¶
Returns a Gaussian Process surrogate model.
- Parameters:
scenario (Scenario) –
model_type (str | None, defaults to None) – Which gaussian process model should be chosen. Choose between vanilla and mcmc.
kernel (kernels.Kernel | None, defaults to None) – The kernel used in the surrogate model.
- Returns:
model – The instantiated gaussian process.
- Return type:
- static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]¶
Returns the mean aggregation strategy for the multi objective algorithm.
- Parameters:
scenario (Scenario) –
objective_weights (list[float] | None, defaults to None) – Weights for an weighted average. Must be of the same length as the number of objectives.
- Return type:
- static get_random_design(scenario, *, probability=0.08447232371720552)[source]¶
Returns
ProbabilityRandomDesign
for interleaving configurations.- Parameters:
probability (float, defaults to 0.08447232371720552) – Probability that a configuration will be drawn at random.
- Return type:
- class smac.facade.HyperbandFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
RandomFacade
Facade to use model-free Hyperband [LJDR18] for algorithm configuration.
Uses Random Aggressive Online Racing (ROAR) to compare configurations, a random initial design and the Hyperband intensifier.
- static get_intensifier(scenario, *, min_challenger=1, eta=3)[source]¶
Returns a Hyperband intensifier instance. That means that budgets are supported.
- min_challengerint, defaults to 1
Minimal number of challengers to be considered (even if time_bound is exhausted earlier).
- etafloat, defaults to 3
The “halving” factor after each iteration in a Successive Halving run.
- Return type:
- class smac.facade.HyperparameterOptimizationFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
AbstractFacade
- static get_acquisition_function(scenario, *, xi=0.0)[source]¶
Returns an Expected Improvement acquisition function.
- static get_acquisition_maximizer(scenario, *, challengers=10000, local_search_iterations=10)[source]¶
Returns local and sorted random search as acquisition maximizer.
Warning
If you experience RAM issues, try to reduce the number of challengers.
- Parameters:
challengers (int, defaults to 10000) – Number of challengers.
local_search_iterations (int, defauts to 10) – Number of local search iterations.
- Return type:
- static get_initial_design(scenario, *, n_configs=None, n_configs_per_hyperparamter=10, max_ratio=0.1, additional_configs=[])[source]¶
Returns a Sobol design instance.
- Parameters:
scenario (Scenario) –
n_configs (int | None, defaults to None) – Number of initial configurations (disables the arguments
n_configs_per_hyperparameter
).n_configs_per_hyperparameter (int, defaults to 10) – Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and
n_configs_per_hyperparameter
is set to 10, then 50 initial configurations will be samples.max_ratio (float, defaults to 0.1) – Use at most
scenario.n_trials
*max_ratio
number of configurations in the initial design. Additional configurations are not affected by this parameter.additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.
- Return type:
- static get_intensifier(scenario, *, min_challenger=1, min_config_calls=1, max_config_calls=3, intensify_percentage=1e-10)[source]¶
Returns
Intensifier
as intensifier. Uses the default configuration forrace_against
.- Parameters:
scenario (Scenario) –
min_config_calls (int, defaults to 1) – Minimum number of trials per config (summed over all calls to intensify).
max_config_calls (int, defaults to 1) – Maximum number of trials per config (summed over all calls to intensify).
min_challenger (int, defaults to 3) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).
intensify_percentage (float, defaults to 1e-10) – How much percentage of the time should configurations be intensified (evaluated on higher budgets or more instances). This parameter is accessed in the SMBO class.
- Return type:
- static get_model(scenario, *, n_trees=10, ratio_features=1.0, min_samples_split=2, min_samples_leaf=1, max_depth=1048576, bootstrapping=True)[source]¶
Returns a random forest as surrogate model.
- Parameters:
n_trees (int, defaults to 10) – The number of trees in the random forest.
ratio_features (float, defaults to 5.0 / 6.0) – The ratio of features that are considered for splitting.
min_samples_split (int, defaults to 3) – The minimum number of data points to perform a split.
min_samples_leaf (int, defaults to 3) – The minimum number of data points in a leaf.
max_depth (int, defaults to 20) – The maximum depth of a single tree.
bootstrapping (bool, defaults to True) – Enables bootstrapping.
- Return type:
- static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]¶
Returns the mean aggregation strategy for the multi objective algorithm.
- Parameters:
scenario (Scenario) –
objective_weights (list[float] | None, defaults to None) – Weights for an weighted average. Must be of the same length as the number of objectives.
- Return type:
- static get_random_design(scenario, *, probability=0.2)[source]¶
Returns
ProbabilityRandomDesign
for interleaving configurations.- Parameters:
probability (float, defaults to 0.2) – Probability that a configuration will be drawn at random.
- Return type:
- class smac.facade.MultiFidelityFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
HyperparameterOptimizationFacade
This facade configures SMAC in a multi-fidelity setting. The way this facade combines the components is the following and exploits fidelity information in the following form:
The initial design is a RandomInitialDesign.
2. The intensification is Hyperband. The configurations from the initial design are presented as challengers and executed in the Hyperband fashion. 3. The model is a RandomForest surrogate model. The data to train it is collected by
SMBO._collect_data
.Notably, the method searches through the runhistory and collects the data from the highest fidelity level, that supports at least
SMBO._min_samples_model
number of configurations.The acquisition function is
EI
, presenting the value of a candidate configuration.The acquisition optimizer is
LocalAndSortedRandomSearch
. It optimizes the acquisition
function to present the best configuration as challenger to the intensifier. From now on 2. works as follows: The intensifier runs the challenger in a Hyperband fashion against the existing configurations and their observed performances until the challenger does not survive a fidelity level. The intensifier can inquire about a known configuration on a yet unseen fidelity if necessary.
The loop 2-5 continues until a termination criterion is reached.
Note
For intensification the data acquisition and aggregation strategy in step 2 is changed. Incumbents are updated by the mean performance over the intersection of instances, that the challenger and incumbent have in common (
abstract_intensifier._compare_configs
). The model in step 3 is trained on all the available instance performance values. The datapoints for a hyperparameter configuration are disambiguated by the instance features or an index as replacement if no instance features are available.- static get_initial_design(scenario, *, n_configs=None, n_configs_per_hyperparamter=10, max_ratio=0.1, additional_configs=[])[source]¶
Returns a random initial design.
- Parameters:
scenario (Scenario) –
n_configs (int | None, defaults to None) – Number of initial configurations (disables the arguments
n_configs_per_hyperparameter
).n_configs_per_hyperparameter (int, defaults to 10) – Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and
n_configs_per_hyperparameter
is set to 10, then 50 initial configurations will be samples.max_ratio (float, defaults to 0.1) – Use at most
scenario.n_trials
*max_ratio
number of configurations in the initial design. Additional configurations are not affected by this parameter.additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.
- Return type:
- static get_intensifier(scenario, *, eta=3, min_challenger=1, n_seeds=1)[source]¶
Returns a Hyperband intensifier instance. That means that budgets are supported.
- min_challengerint, defaults to 1
Minimal number of challengers to be considered (even if time_bound is exhausted earlier).
- etafloat, defaults to 3
The “halving” factor after each iteration in a Successive Halving run.
- n_seedsint | None, defaults to None
The number of seeds to use if the target function is non-deterministic.
- Return type:
- class smac.facade.RandomFacade(scenario, target_function, *, model=None, acquisition_function=None, acquisition_maximizer=None, initial_design=None, random_design=None, intensifier=None, multi_objective_algorithm=None, runhistory_encoder=None, logging_level=None, callbacks=[], overwrite=False)[source]¶
Bases:
AbstractFacade
Facade to use Random Online Aggressive Racing (ROAR).
Aggressive Racing: When we have a new configuration θ, we want to compare it to the current best configuration, the incumbent θ*. ROAR uses the ‘racing’ approach, where we run few times for unpromising θ and many times for promising configurations. Once we are confident enough that θ is better than θ*, we update the incumbent θ* ⟵ θ. Aggressive means rejecting low-performing configurations very early, often after a single run. This together is called aggressive racing.
ROAR Loop: The main ROAR loop looks as follows:
Select a configuration θ uniformly at random.
Compare θ to incumbent θ* online (one θ at a time):
Reject/accept θ with aggressive racing
Setup: Uses a random model and random search for the optimization of the acquisition function.
Note
The surrogate model and the acquisition function is not used during the optimization and therefore replaced by dummies.
- static get_acquisition_function(scenario)[source]¶
The random facade is not using an acquisition function. Therefore, we simply return a dummy function.
- Return type:
- static get_acquisition_maximizer(scenario)[source]¶
We return
RandomSearch
as maximizer which samples configurations randomly from the configuration space and therefore neither uses the acquisition function nor the model.- Return type:
- static get_initial_design(scenario, *, additional_configs=[])[source]¶
Returns an initial design, which returns the default configuration.
- Parameters:
additional_configs (list[Configuration], defaults to []) – Adds additional configurations to the initial design.
- Return type:
- static get_intensifier(scenario, *, min_challenger=1, min_config_calls=1, max_config_calls=3, intensify_percentage=0.5)[source]¶
Returns
Intensifier
as intensifier. Uses the default configuration forrace_against
.Note
Please use
HyperbandFacade
if you want to incorporate budgets.Warning
If you are in an algorithm configuration setting, consider increasing
max_config_calls
.- Return type:
- static get_model(scenario)[source]¶
The model is used in the acquisition function. Since we do not use an acquisition function, we return a dummy model (returning random values in this case).
- Return type:
- static get_multi_objective_algorithm(scenario, *, objective_weights=None)[source]¶
Returns the mean aggregation strategy for the multi objective algorithm.
- Parameters:
scenario (Scenario) –
objective_weights (list[float] | None, defaults to None) – Weights for an weighted average. Must be of the same length as the number of objectives.
- Return type:
- static get_random_design(scenario)[source]¶
Just like the acquisition function, we do not use a random design. Therefore, we return a dummy design.
- Return type: