smac.intensifier

Interfaces

class smac.intensifier.AbstractIntensifier(scenario, n_seeds=None, max_config_calls=None, max_incumbents=10, seed=None)[source]

Bases: object

Abstract implementation of an intensifier supporting multi-fidelity, multi-objective, and multi-threading. The abstract intensifier keeps track of the incumbent, which is updated everytime the runhistory changes.

Parameters:

  • n_seeds (int | None, defaults to None) – How many seeds to use for each instance. It is used in the abstract intensifier to determine validation trials.

  • max_config_calls (int, defaults to None) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be max evaluated for a configuration. It is used in the abstract intensifier to determine validation trials.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

  • seed (int, defaults to None) – Internal seed used for random events like shuffle seeds.

abstract __iter__()[source]

Main loop of the intensifier. This method always returns a TrialInfo object, although the intensifier algorithm may need to wait for the result of the trial. Please refer to a specific intensifier to get more information.

Return type:

Iterator[TrialInfo]

__post_init__()[source]

Fills self._tf_seeds and self._tf_instances. Moreover, the incumbents are updated.

Return type:

None

property config_generator: Iterator[Configuration]

Based on the configuration selector, an iterator is returned that generates configurations.

property config_selector: ConfigSelector

The configuration selector for the intensifier.

get_callback()[source]

The intensifier makes use of a callback to efficiently update the incumbent based on the runhistory (every time new information is available). Moreover, incorporating the callback here allows developers more options in the future.

Return type:

Callback

get_incumbent()[source]

Returns the current incumbent in a single-objective setting.

Return type:

Configuration | None

get_incumbent_instance_seed_budget_key_differences(compare=False)[source]

There are situations in which incumbents are evaluated on more trials than others. This method returns the instances that are not part of the lowest intersection of instances for all incumbents.

Return type:

list[InstanceSeedBudgetKey]

get_incumbent_instance_seed_budget_keys(compare=False)[source]

Find the lowest intersection of instance-seed-budget keys for all incumbents.

Return type:

list[InstanceSeedBudgetKey]

get_incumbents(sort_by=None)[source]

Returns the incumbents (points on the pareto front) of the runhistory as copy. In case of a single-objective optimization, only one incumbent (if is) is returned.

Return type:

list[Configuration]

Returns:

  • configs (list[Configuration]) – The configs of the Pareto front.

  • sort_by (str, defaults to None) – Sort the trials by cost (lowest cost first) or num_trials (config with lowest number of trials first).

get_instance_seed_budget_keys(config, compare=False)[source]

Returns the instance-seed-budget keys for a given configuration. This method is used for updating the incumbents and might differ for different intensifiers. For example, if incumbents should only be compared on the highest observed budgets.

Return type:

list[InstanceSeedBudgetKey]

get_instance_seed_keys_of_interest(*, validate=False, seed=None)[source]

Returns a list of instance-seed keys. Considers seeds and instances from the runhistory (self._tf_seeds and self._tf_instances). If no seeds or instances were found, new seeds and instances are generated based on the global intensifier seed.

Warning

The passed seed is only used for validation. For training, the global intensifier seed is used.

Parameters:

  • validate (bool, defaults to False) – Whether to get validation trials or training trials. The only difference lies in different seeds.

  • seed (int | None, defaults to None) – The seed used for the validation trials.

Returns:

instance_seed_keys – Instance-seed keys of interest.

Return type:

list[InstanceSeedKey]

get_rejected_configs()[source]

Returns rejected configurations when racing against the incumbent failed.

Return type:

list[Configuration]

get_state()[source]

The current state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

dict[str, Any]

get_trials_of_interest(config, *, validate=False, seed=None)[source]

Returns the trials of interest for a given configuration. Expands the keys from get_instance_seed_keys_of_interest with the config.

Return type:

list[TrialInfo]

property incumbents_changed: int

How often the incumbents have changed.

load(filename)[source]

Loads the latest state of the intensifier including the incumbents and trajectory.

Return type:

None

property meta: dict[str, Any]

Returns the meta data of the created object.

reset()[source]

Reset the internal variables of the intensifier.

Return type:

None

property runhistory: RunHistory

Runhistory of the intensifier.

save(filename)[source]

Saves the current state of the intensifier. In addition to the state (retrieved by get_state), this method also saves the incumbents and trajectory.

Return type:

None

set_state(state)[source]

Sets the state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

None

property trajectory: list[TrajectoryItem]

Returns the trajectory (changes of incumbents) of the optimization run.

update_incumbents(config)[source]

Updates the incumbents. This method is called everytime a trial is added to the runhistory. Since only the affected config and the current incumbents are used, this method is very efficient. Furthermore, a configuration is only considered incumbent if it has a better performance on all incumbent instances.

Crucially, if there is no incumbent (at the start) then, the first configuration assumes incumbent status. For the next configuration, we need to check if the configuration is better on all instances that have been evaluated for the incumbent. If this is the case, then we can replace the incumbent. Otherwise, a) we need to requeue the config to obtain the missing instance-seed-budget combination or b) mark this configuration as inferior (“rejected”) to not consider it again. The comparison behaviour is controlled by self.get_instance_seed_budget_keys() and self.get_incumbent_instance_seed_budget_keys().

Notably, this method is written to support both multi-fidelity and multi-objective optimization. While the get_instance_seed_budget_keys() method and self.get_incumbent_instance_seed_budget_keys() are used for the multi-fidelity behaviour, calculate_pareto_front() is used as a hard coded way to support multi-objective optimization, including the single objective as special case. calculate_pareto_front() is called on the set of all (in case of MO) incumbents amended with the challenger configuration, provided it has a sufficient overlap in seed-instance-budget combinations.

Lastly, if we have a self._max_incumbents and the pareto front provides more than this specified amount, we cut the incumbents using crowding distance.

Return type:

None

property used_walltime: float

Returns used wallclock time.

abstract property uses_budgets: bool

If the intensifier needs to make use of budgets.

abstract property uses_instances: bool

If the intensifier needs to make use of instances.

abstract property uses_seeds: bool

If the intensifier needs to make use of seeds.

class smac.intensifier.Hyperband(scenario, eta=3, n_seeds=1, instance_seed_order='shuffle_once', max_incumbents=10, incumbent_selection='highest_observed_budget', seed=None)[source]

Bases: SuccessiveHalving

See SuccessiveHalving for documentation.

get_state()[source]

The current state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

dict[str, Any]

reset()[source]

Resets the internal variables of the intensifier, including the tracker and the next bracket.

Return type:

None

set_state(state)[source]

Sets the state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

None

class smac.intensifier.Intensifier(scenario, max_config_calls=3, max_incumbents=10, retries=16, seed=None)[source]

Bases: AbstractIntensifier

Implementation of an intensifier supporting multi-fidelity, multi-objective, and multi-processing. Races challengers against current incumbents.

The behaviour of this intensifier is as follows:

  • First, adds configs from the runhistory to the queue with N=1 (they will be ignored if they are already evaluated).

  • While loop:

    • If queue is empty: Intensifies exactly one more instance of one incumbent and samples a new configuration afterwards.

    • If queue is not empty: Configs in the queue are evaluated on N=(N*2) instances if they might be better than the incumbents. If not, they are removed from the queue and rejected forever.

Parameters:

  • max_config_calls (int, defaults to 3) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be maxed evaluated for a configuration.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

  • retries (int, defaults to 16) – How many more iterations should be done in case no new trial is found.

  • seed (int, defaults to None) – Internal seed used for random events, like shuffle seeds.

__iter__()[source]

This iter method holds the logic for the intensification loop. Some facts about the loop:

  • Adds existing configurations from the runhistory to the queue (that means it supports user-inputs).

  • Everytime an incumbent (with the lowest amount of trials) is intensified, a new challenger is added to the queue.

  • If all incumbents are evaluated on the same trials, a new trial is added to one of the incumbents.

  • Only challengers which are not rejected/running/incumbent are intensified by N*2.

Returns:

trials – Iterator over the trials.

Return type:

Iterator[TrialInfo]

get_state()[source]

The current state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

dict[str, Any]

reset()[source]

Resets the internal variables of the intensifier including the queue.

Return type:

None

set_state(state)[source]

Sets the state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

None

property uses_budgets: bool

If the intensifier needs to make use of budgets.

property uses_instances: bool

If the intensifier needs to make use of instances.

property uses_seeds: bool

If the intensifier needs to make use of seeds.

class smac.intensifier.SuccessiveHalving(scenario, eta=3, n_seeds=1, instance_seed_order='shuffle_once', max_incumbents=10, incumbent_selection='highest_observed_budget', seed=None)[source]

Bases: AbstractIntensifier

Implementation of Succesive Halving supporting multi-fidelity, multi-objective, and multi-processing. Internally, a tracker keeps track of configurations and their bracket and stage.

The behaviour of this intensifier is as follows:

  • First, adds configurations from the runhistory to the tracker. The first stage is always filled-up. For example, the user provided 4 configs with the tell-method but the first stage requires 8 configs: 4 new configs are sampled and added together with the provided configs as a group to the tracker.

  • While loop:

    • If a trial in the tracker has not been yielded yet, yield it.

    • If we are running out of trials, we simply add a new batch of configurations to the first stage.

Note

The implementation natively supports brackets from Hyperband. However, in the case of Successive Halving, only one bracket is used.

Parameters:

  • eta (int, defaults to 3) – Input that controls the proportion of configurations discarded in each round of Successive Halving.

  • n_seeds (int, defaults to 1) – How many seeds to use for each instance.

  • instance_seed_order (str, defaults to "shuffle_once") –

    How to order the instance-seed pairs. Can be set to:

    • None: No shuffling at all and use the instance-seed order provided by the user.

    • shuffle_once: Shuffle the instance-seed keys once and use the same order across all runs.

    • shuffle: Shuffles the instance-seed keys for each bracket individually.

  • incumbent_selection (str, defaults to "highest_observed_budget") –

    How to select the incumbent when using budgets. Can be set to:

    • any_budget: Incumbent is the best on any budget i.e., best performance regardless of budget.

    • highest_observed_budget: Incumbent is the best in the highest budget run so far.

    • highest_budget: Incumbent is selected only based on the highest budget.

  • max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

  • seed (int, defaults to None) – Internal seed used for random events like shuffle seeds.

__post_init__()[source]

Post initialization steps after the runhistory has been set.

Return type:

None

get_instance_seed_budget_keys(config, compare=False)[source]

Returns the instance-seed-budget keys for a given configuration. This method supports highest_budget, which only returns the instance-seed-budget keys for the highest budget (if specified). In this case, the incumbents in update_incumbents are only changed if the costs on the highest budget are lower.

Parameters:

  • config (Configuration) – The Configuration to be queried

  • compare (bool, defaults to False) – Get rid of the budget information for comparing if the configuration was evaluated on the same instance-seed keys.

Return type:

list[InstanceSeedBudgetKey]

get_state()[source]

The current state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

dict[str, Any]

get_trials_of_interest(config, *, validate=False, seed=None)[source]

Returns the trials of interest for a given configuration. Expands the keys from get_instance_seed_keys_of_interest with the config.

Return type:

list[TrialInfo]

property meta: dict[str, Any]

Returns the meta data of the created object.

print_tracker()[source]

Prints the number of configurations in each bracket/stage.

Return type:

None

reset()[source]

Reset the internal variables of the intensifier including the tracker.

Return type:

None

set_state(state)[source]

Sets the state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

None

property uses_budgets: bool

If the intensifier needs to make use of budgets.

property uses_instances: bool

If the intensifier needs to make use of instances.

property uses_seeds: bool

If the intensifier needs to make use of seeds.

Modules

smac.intensifier.abstract_intensifier

smac.intensifier.hyperband

smac.intensifier.intensifier

smac.intensifier.successive_halving

previous

smac.acquisition.maximizer.random_search

next

smac.intensifier.abstract_intensifier