smac.intensifier

Interfaces

class smac.intensifier.AbstractIntensifier(scenario, min_config_calls=1, max_config_calls=2000, min_challenger=1, seed=None)[source]

Bases: object

Base class for all racing methods.

The intensification is designed to output a TrialInfo object with enough information to run a given configuration (for example, the trial info contains the instance/seed pair, as well as the associated resources).

A worker can execute this TrialInfo object and produce a TrialValue object with the execution results. Each intensifier process the TrialValue object and updates its internal state in preparation for the next iteration.

Parameters:

  • scenario (Scenario) –

  • min_config_calls (int, defaults to 1) – Minimum number of trials per config (summed over all calls to intensify).

  • max_config_calls (int, defaults to 2000) – Maximum number of trials per config (summed over all calls to intensify).

  • min_challenger (int, defaults to 1) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).

  • seed (int | None, defaults to none) –

get_next_trial(challengers, incumbent, get_next_configurations, runhistory, repeat_configs=True, n_workers=1)[source]

Abstract method for choosing the next challenger. If no more challengers are available, the method should issue a SKIP via RunInfoIntent.SKIP, so that a new iteration can sample new configurations.

Parameters:

  • challengers (list[Configuration] | None) – Promising configurations.

  • incumbent (Configuration) – Incumbent configuration.

  • get_next_configurations (Callable[[], Iterator[Configuration]] | None, defaults to none) – Function that generates next configurations to use for racing.

  • runhistory (RunHistory) –

  • repeat_configs (bool, defaults to true) – If false, an evaluated configuration will not be generated again.

  • n_workers (int, optional, defaults to 1) – The maximum number of workers available.

Return type:

tuple[TrialInfoIntent, TrialInfo]

Returns:

  • TrialInfoIntent – Indicator of how to consume the TrialInfo object.

  • TrialInfo – An object that encapsulates necessary information of the trial.

abstract get_target_function_budgets()[source]

Which budgets are used to call the target function.

Return type:

list[Optional[float]]

abstract get_target_function_instances()[source]

Which instances are used to call the target function.

Return type:

list[Optional[str]]

abstract get_target_function_seeds()[source]

Which seeds are used to call the target function.

Return type:

list[int]

property iteration_done: bool

Whether an iteration is done or not.

Return type:

bool

property meta: dict[str, Any]

Returns the meta data of the created object.

Return type:

dict[str, Any]

property num_trials: int

How many trials have been done in an iteration so far.

Return type:

int

process_results(trial_info, trial_value, incumbent, runhistory, time_bound, log_trajectory=True)[source]

The intensifier stage will be updated based on the results/status of a configuration execution. Also, a incumbent will be determined.

Parameters:

  • trial_info (TrialInfo) –

  • trial_value (TrialValue) –

  • incumbent (Configuration | None) – Best configuration seen so far.

  • runhistory (RunHistory) –

  • time_bound (float) – Time [sec] available to perform intensify.

  • log_trajectory (bool) – Whether to log changes of incumbents in the trajectory.

Return type:

tuple[Configuration, float | list[float]]

Returns:

  • incumbent (Configuration) – Current (maybe new) incumbent configuration.

  • incumbent_costs (float | list[float]) – Empirical cost(s) of the incumbent configuration.

property repeat_configs: bool

Whether configs should be repeated or not.

Return type:

bool

abstract property uses_budgets: bool

If the intensifier needs to make use of budgets.

Return type:

bool

abstract property uses_instances: bool

If the intensifier needs to make use of instances.

Return type:

bool

abstract property uses_seeds: bool

If the intensifier needs to make use of seeds.

Return type:

bool

class smac.intensifier.AbstractParallelIntensifier(scenario, min_challenger=1, seed=None)[source]

Bases: AbstractIntensifier

Common Racer class for Intensifiers that will schedule configurations on a parallel fashion.

This class instantiates intensifier objects on a need basis, that is, to prevent workers from being idle. This intensifier objects will give configurations to run.

scenario : Scenario min_challenger : int, optional, defaults to 1

Minimum number of trials per config (summed over all calls to intensify).

seed : int | None, defaults to None

get_next_trial(challengers, incumbent, get_next_configurations, runhistory, repeat_configs=False, n_workers=1)[source]

This procedure decides from which instance to pick a config, in order to determine the next run. To prevent having idle workers, this procedure creates new instances up to the maximum number of workers available.

If no new intensifier instance can be created and all intensifier objects need to wait for more data, this procedure sends a wait request to smbo.

Parameters:

  • challengers (list[Configuration] | None) – Promising configurations.

  • incumbent (Configuration) – Incumbent configuration.

  • get_next_configurations (Callable[[], Iterator[Configuration]] | None, defaults to none) – Function that generates next configurations to use for racing.

  • runhistory (RunHistory) –

  • repeat_configs (bool, defaults to true) – If false, an evaluated configuration will not be generated again.

  • n_workers (int, optional, defaults to 1) – The maximum number of workers available.

Return type:

tuple[TrialInfoIntent, TrialInfo]

Returns:

  • TrialInfoIntent – Indicator of how to consume the TrialInfo object.

  • TrialInfo – An object that encapsulates necessary information of the trial.

process_results(trial_info, trial_value, incumbent, runhistory, time_bound, log_trajectory=True)[source]

The intensifier stage will be updated based on the results/status of a configuration execution. To do so, this procedures redirects the result argument, to the respective intensifier object that generated the original config.

Also, an incumbent will be determined. This determination is done using the complete run history, so we rely on the current intensifier choice of incumbent. That is, no need to go over each instance to get the incumbent, as there is no local runhistory

Parameters:

  • trial_info (TrialInfo) –

  • trial_value (TrialValue) –

  • incumbent (Configuration | None) – Best configuration seen so far.

  • runhistory (RunHistory) –

  • time_bound (float) – Time [sec] available to perform intensify.

  • log_trajectory (bool) – Whether to log changes of incumbents in the trajectory.

Return type:

tuple[Configuration, float | list[float]]

Returns:

  • incumbent (Configuration) – Current (maybe new) incumbent configuration.

  • incumbent_costs (float | list[float]) – Empirical cost(s) of the incumbent configuration.

property uses_seeds: bool

If the intensifier needs to make use of seeds.

Return type:

bool

class smac.intensifier.Hyperband(scenario, instance_seed_pairs=None, instance_order='shuffle_once', incumbent_selection='highest_executed_budget', n_initial_challengers=None, min_challenger=1, eta=3, seed=None, n_seeds=None)[source]

Bases: SuccessiveHalving

Races multiple challengers against an incumbent using Hyperband method.

Implementation from “BOHB: Robust and Efficient Hyperparameter Optimization at Scale” (Falkner et al. 2018)

Hyperband is an extension of the Successive Halving intensifier. Please refer to SuccessiveHalving documentation for more detailed information about the different types of budgets possible and the way instances are handled.

Internally, this class uses the HyperbandWorker class which actually implements the hyperband logic. To allow for parallelism, Hyperband can create multiple HyperbandWorker instances, based on the number of idle workers available.

Parameters:

  • scenario (Scenario) –

  • instance_seed_pairs (list[tuple[str | None, int]] | None, defaults to None) – This argument is used by Hyperband.

  • instance_order (str | None, defaults to shuffle_once) – How to order the instances. Can be set to: * None: Use as is given by the user. * shuffle_once: Shuffle once and use across all Successive Halving run . * shuffle: Shuffle before every Successive Halving run

  • incumbent_selection (str, defaults to highest_executed_budget) – How to select the incumbent in Successive Halving. Only active for (real-valued) budgets. Can be set to: * highest_executed_budget: Incumbent is the best in the highest budget run so far. * highest_budget: Incumbent is selected only based on the highest budget. * any_budget: Incumbent is the best on any budget i.e., best performance regardless of budget.

  • n_initial_challengers (int | None, defaults to None) – Number of challengers to consider for the initial budget. If not specified, it is calculated internally.

  • min_challenger (int, defaults to 1) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).

  • eta (float, defaults to 3) – The “halving” factor after each iteration in a Successive Halving run.

  • seed (int | None, defaults to None) –

  • n_seeds (int | None, defaults to None) – The number of seeds to use if the target function is non-deterministic.

class smac.intensifier.Intensifier(scenario, min_config_calls=1, max_config_calls=2000, min_challenger=2, intensify_percentage=0.5, race_against=None, seed=None)[source]

Bases: AbstractIntensifier

Races challengers against an incumbent.

SMAC’s intensification procedure, in detail: Intensify(Θ_new, θ_inc, M, R, t_intensify, Π, c_hat) c_hat(θ, Π’) denotes the empirical cost of θ on the subset of instances Π’ ⊆ Π, based on the trials in R; max_config_calls is a parameter where: Θ_new: Sequence of parameter settings to evaluate, challengers in this class. θ_inc: incumbent parameter setting, incumbent in this class.

1 for i := 1, … , length(Θ_new) do 2 θ_new ← Θ_new[i]

STAGE -> RUN_INCUMBENT

3 if R contains less than max_config_calls trials with configuration θ_inc then 4 Π’ ← {π’∈ Π | R contains less than or equal number of trials using θ_inc and π’ 0 than using θ_inc and any other π’’∈ Π} 5 π ← instance sampled uniformly at random from Π’ 6 s ← seed, drawn uniformly at random 7 R ← ExecuteRun(R, θ_inc, π, s) 8 N ← 1

STAGE -> RUN_CHALLENGER

9 while true do 10 S_missing ← {instance, seed} pairs for which θ_inc was run before, but not θ_new 11 S_torun ← random subset of S_missing of size min(N, size(S_missing)) 12 foreach (π, s) ∈ S_torun do R ← ExecuteRun(R, θ_new, π, s) 13 S_missing ← S_missing \ S_torun 14 Π_common ← instances for which we previously ran both θ_inc and θ_new 15 if c_hat(θ_new, Π_common) > c_hat(θ_inc, Π_common) then break 16 else if S_missing = ∅ then θ_inc ← θ_new; break 17 else N ← 2 · N 18 if time spent in this call to this procedure exceeds t_intensify and i ≥ 2 then break 19 return [R, θ_inc]

Parameters:

  • scenario (Scenario) –

  • min_config_calls (int, defaults to 1) – Minimum number of trials per config (summed over all calls to intensify).

  • max_config_calls (int, defaults to 2000) – Maximum number of trials per config (summed over all calls to intensify).

  • min_challenger (int, defaults to 2) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).

  • intensify_percentage (float, defaults to 0.5) – How much percentage of the time should configurations be intensified (evaluated on higher budgets or more instances). This parameter is accessed in the SMBO class.

  • race_against (Configuration | None, defaults to none) – If incumbent changes, race this configuration always against new incumbent. Prevents sometimes over-tuning.

  • seed (int | None, defaults to none) –

get_next_challenger(challengers, get_next_configurations)[source]

This function returns the next challenger, that should be exercised though lines 8-17.

It does so by populating _configs_to_run, which is a pool of configuration from which the racer will sample. Each configuration within _configs_to_run, will be intensified on different instances/seed registered in self._to_run as stated in line 11.

A brand new configuration should only be sampled, after all self._to_run instance seed pairs are exhausted.

This method triggers a call to _next_iteration if there are no more configurations to run, for the current intensification loop. This marks the transition to Line 2, where a new configuration to intensify will be drawn from epm/initial challengers.

Parameters:

  • challengers (list[Configuration] | None, defaults to None) – Promising configurations.

  • get_next_configurations (Callable[[], Iterator[Configuration]] | None) – Function that generates next configurations to use for racing.

Return type:

tuple[Optional[Configuration], bool]

Returns:

  • configuration (Configuration | None) – The configuration of the selected challenger.

  • new_challenger (bool) – If the configuration is a new challenger.

get_next_trial(challengers, incumbent, get_next_configurations, runhistory, repeat_configs=True, n_workers=1)[source]

This procedure is in charge of generating a TrialInfo object to comply with lines 7 (in case stage is stage == RUN_INCUMBENT) or line 12 (In case of stage == RUN_CHALLENGER).

A TrialInfo object encapsulates the necessary information for a worker to execute the job, nevertheless, a challenger is not always available. This could happen because no more configurations are available or the new configuration to try was already executed.

To circumvent this, an intent is also returned:

  • intent == RUN: Run the TrialInfo object (Normal Execution).

  • intent == SKIP: Skip this iteration. No challenger is available. In particular because challenger is the same

    as incumbent

Parameters:

  • challengers (list[Configuration] | None) – Promising configurations.

  • incumbent (Configuration) – Incumbent configuration.

  • get_next_configurations (Callable[[], Iterator[Configuration]] | None, defaults to none) – Function that generates next configurations to use for racing.

  • runhistory (RunHistory) –

  • repeat_configs (bool, defaults to true) – If false, an evaluated configuration will not be generated again.

  • n_workers (int, optional, defaults to 1) – The maximum number of workers available.

Return type:

tuple[TrialInfoIntent, TrialInfo]

Returns:

  • TrialInfoIntent – Indicator of how to consume the TrialInfo object.

  • TrialInfo – An object that encapsulates necessary information of the trial.

get_target_function_budgets()[source]

Which budgets are used to call the target function.

Return type:

list[Optional[float]]

get_target_function_instances()[source]

Which instances are used to call the target function.

Return type:

list[Optional[str]]

get_target_function_seeds()[source]

Which seeds are used to call the target function.

Return type:

list[int]

property intensify_percentage: float

How much percentage of the time should configurations be intensified (evaluated on higher budgets or more instances). This parameter is accessed in the SMBO class.

Return type:

float

property meta: dict[str, Any]

Returns the meta data of the created object.

Return type:

dict[str, Any]

process_results(trial_info, trial_value, incumbent, runhistory, time_bound, log_trajectory=True)[source]

The intensifier stage will be updated based on the results/status of a configuration execution. During intensification, the following can happen:

  • Challenger raced against incumbent.

  • Also, during a challenger run, a capped exception can be triggered, where no racer post processing is needed.

  • A run on the incumbent for more confidence needs to be processed, IntensifierStage.PROCESS_INCUMBENT_RUN.

  • The first run results need to be processed (PROCESS_FIRST_CONFIG_RUN).

At the end of any run, checks are done to move to a new iteration.

Parameters:

  • trial_info (TrialInfo) –

  • trial_value (TrialValue) –

  • incumbent (Configuration | None) – Best configuration seen so far.

  • runhistory (RunHistory) –

  • time_bound (float) – Time [sec] available to perform intensify.

  • log_trajectory (bool) – Whether to log changes of incumbents in the trajectory.

Return type:

tuple[Configuration, float]

Returns:

  • incumbent (Configuration) – Current (maybe new) incumbent configuration.

  • incumbent_costs (float | list[float]) – Empirical cost(s) of the incumbent configuration.

property uses_budgets: bool

If the intensifier needs to make use of budgets.

Return type:

bool

property uses_instances: bool

If the intensifier needs to make use of instances.

Return type:

bool

property uses_seeds: bool

If the intensifier needs to make use of seeds.

Return type:

bool

class smac.intensifier.SuccessiveHalving(scenario, instance_seed_pairs=None, instance_order='shuffle_once', incumbent_selection='highest_executed_budget', n_initial_challengers=None, min_challenger=1, eta=3, seed=None, n_seeds=None)[source]

Bases: AbstractParallelIntensifier

Races multiple challengers against an incumbent using Successive Halving method.

Implementation following the description in “BOHB: Robust and Efficient Hyperparameter Optimization at Scale” (Falkner et al. 2018) Supplementary reference: http://proceedings.mlr.press/v80/falkner18a/falkner18a-supp.pdf

Successive Halving intensifier (and Hyperband) can operate on two kinds of budgets:

  1. Instances as budget:

When multiple instances are provided or when run objective is “runtime”, this is the criterion used as budget for successive halving iterations i.e., the budget determines how many instances the challengers are evaluated on at a time. Top challengers for the next iteration are selected based on the combined performance across all instances used. If min_budget and max_budget are not provided, then they are set to 1 and total number of available instances respectively by default.

  1. Real-valued budget:

This is used when there is only one instance provided and when run objective is “quality”, i.e., budget is a positive, real-valued number that can be passed to the target function as an argument. It can be used to control anything by the target function, Eg: number of epochs for training a neural network. The parameters min_budget and max_budget are required for this type of budget.

This class instantiates SuccessiveHalvingWorker objects on a need basis, that is, to prevent workers from being idle. The actual logic that implements the Successive halving method lies in the worker class.

Parameters:

  • scenario (Scenario) –

  • instance_seed_pairs (list[tuple[str | None, int]] | None, defaults to None) – This argument is used by Hyperband.

  • instance_order (str | None, defaults to shuffle_once) – How to order the instances. Can be set to: * None: Use as is given by the user. * shuffle_once: Shuffle once and use across all Successive Halving run . * shuffle: Shuffle before every Successive Halving run

  • incumbent_selection (str, defaults to highest_executed_budget) – How to select the incumbent in Successive Halving. Only active for (real-valued) budgets. Can be set to: * highest_executed_budget: Incumbent is the best in the highest budget run so far. * highest_budget: Incumbent is selected only based on the highest budget. * any_budget: Incumbent is the best on any budget i.e., best performance regardless of budget.

  • n_initial_challengers (int | None, defaults to None) – Number of challengers to consider for the initial budget. If not specified, it is calculated internally.

  • min_challenger (int, defaults to 1) – Minimal number of challengers to be considered (even if time_bound is exhausted earlier).

  • eta (float, defaults to 3) – The “halving” factor after each iteration in a Successive Halving run.

  • seed (int | None, defaults to None) – This seed is not the seed used for target function evaluation but rather for sampling next candidates.

  • n_seeds (int | None, defaults to None) – The number of seeds to use if the target function is non-deterministic.

calculate_budgets(min_budget, max_budget)[source]

Calculates the budgets for successive halving.

Parameters:

  • min_budget (float) – The minimal budget.

  • max_budget (float) – The maximal budget.

Return type:

tuple[int, int, list[float]]

Returns:

  • max_iterations (int) – How many iterations will be performed.

  • n_initial_challengers (int) – How many challengers will be sampled in the first iteration.

  • budgets (list[float]) – The budgets for each iteration.

get_target_function_budgets()[source]

Which budgets are used to call the target function.

Return type:

list[Optional[float]]

get_target_function_instances()[source]

Which instances are used to call the target function.

Return type:

list[Optional[str]]

get_target_function_seeds()[source]

Which seeds are used to call the target function.

Return type:

list[int]

property meta: dict[str, Any]

Returns the meta data of the created object.

Return type:

dict[str, Any]

property uses_budgets: bool

If the intensifier needs to make use of budgets.

Return type:

bool

property uses_instances: bool

If the intensifier needs to make use of instances.

Return type:

bool

Modules

smac.intensifier.abstract_intensifier

smac.intensifier.abstract_parallel_intensifier

smac.intensifier.hyperband

smac.intensifier.hyperband_worker

smac.intensifier.intensifier

smac.intensifier.stages

smac.intensifier.successive_halving

smac.intensifier.successive_halving_worker

previous

smac.acquisition.maximizer.random_search

next

smac.intensifier.abstract_intensifier