smac.intensifier.abstract_intensifier

Classes

AbstractIntensifier(scenario[, n_seeds, ...])

Abstract implementation of an intensifier supporting multi-fidelity, multi-objective, and multi-threading.

Interfaces

class smac.intensifier.abstract_intensifier.AbstractIntensifier(scenario, n_seeds=None, max_config_calls=None, max_incumbents=10, seed=None)

Bases: object

Abstract implementation of an intensifier supporting multi-fidelity, multi-objective, and multi-threading. The abstract intensifier keeps track of the incumbent, which is updated everytime the runhistory changes.

Parameters:

• n_seeds (int | None, defaults to None) – How many seeds to use for each instance. It is used in the abstract intensifier to determine validation trials.

• max_config_calls (int, defaults to None) – Maximum number of configuration evaluations. Basically, how many instance-seed keys should be max evaluated for a configuration. It is used in the abstract intensifier to determine validation trials.

• max_incumbents (int, defaults to 10) – How many incumbents to keep track of in the case of multi-objective.

• seed (int, defaults to None) – Internal seed used for random events like shuffle seeds.

abstract __iter__()

Main loop of the intensifier. This method always returns a TrialInfo object, although the intensifier algorithm may need to wait for the result of the trial. Please refer to a specific intensifier to get more information.

Return type:

Iterator[TrialInfo]

__post_init__()

Fills self._tf_seeds and self._tf_instances. Moreover, the incumbents are updated.

Return type:

None

property config_generator: Iterator[Configuration]

Based on the configuration selector, an iterator is returned that generates configurations.

property config_selector: ConfigSelector

The configuration selector for the intensifier.

get_callback()

The intensifier makes use of a callback to efficiently update the incumbent based on the runhistory (every time new information is available). Moreover, incorporating the callback here allows developers more options in the future.

Return type:

Callback

get_incumbent()

Returns the current incumbent in a single-objective setting.

Return type:

Optional[Configuration]

get_incumbent_instance_seed_budget_key_differences(compare=False)

There are situations in which incumbents are evaluated on more trials than others. This method returns the instances that are not part of the lowest intersection of instances for all incumbents.

Return type:

list[InstanceSeedBudgetKey]

get_incumbent_instance_seed_budget_keys(compare=False)

Find the lowest intersection of instance-seed-budget keys for all incumbents.

Return type:

list[InstanceSeedBudgetKey]

get_incumbents(sort_by=None)

Returns the incumbents (points on the pareto front) of the runhistory as copy. In case of a single-objective optimization, only one incumbent (if is) is returned.

Return type:

list[Configuration]

Returns:

• configs (list[Configuration]) – The configs of the Pareto front.

• sort_by (str, defaults to None) – Sort the trials by cost (lowest cost first) or num_trials (config with lowest number of trials first).

get_instance_seed_budget_keys(config, compare=False)

Returns the instance-seed-budget keys for a given configuration. This method is used for updating the incumbents and might differ for different intensifiers. For example, if incumbents should only be compared on the highest observed budgets.

Return type:

list[InstanceSeedBudgetKey]

get_instance_seed_keys_of_interest(*, validate=False, seed=None)

Returns a list of instance-seed keys. Considers seeds and instances from the runhistory (self._tf_seeds and self._tf_instances). If no seeds or instances were found, new seeds and instances are generated based on the global intensifier seed.

Warning

The passed seed is only used for validation. For training, the global intensifier seed is used.

Parameters:

• validate (bool, defaults to False) – Whether to get validation trials or training trials. The only difference lies in different seeds.

• seed (int | None, defaults to None) – The seed used for the validation trials.

Returns:

instance_seed_keys – Instance-seed keys of interest.

Return type:

list[InstanceSeedKey]

get_rejected_configs()

Returns rejected configurations when racing against the incumbent failed.

Return type:

list[Configuration]

get_state()

The current state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

dict[str, Any]

get_trials_of_interest(config, *, validate=False, seed=None)

Returns the trials of interest for a given configuration. Expands the keys from get_instance_seed_keys_of_interest with the config.

Return type:

list[TrialInfo]

property incumbents_changed: int

How often the incumbents have changed.

load(filename)

Loads the latest state of the intensifier including the incumbents and trajectory.

Return type:

None

property meta: dict[str, Any]

Returns the meta data of the created object.

reset()

Reset the internal variables of the intensifier.

Return type:

None

property runhistory: RunHistory

Runhistory of the intensifier.

save(filename)

Saves the current state of the intensifier. In addition to the state (retrieved by get_state), this method also saves the incumbents and trajectory.

Return type:

None

set_state(state)

Sets the state of the intensifier. Used to restore the state of the intensifier when continuing a run.

Return type:

None

property trajectory: list[smac.runhistory.dataclasses.TrajectoryItem]

Returns the trajectory (changes of incumbents) of the optimization run.

update_incumbents(config)

Updates the incumbents. This method is called everytime a trial is added to the runhistory. Since only the affected config and the current incumbents are used, this method is very efficient. Furthermore, a configuration is only considered incumbent if it has a better performance on all incumbent instances.

Crucially, if there is no incumbent (at the start) then, the first configuration assumes incumbent status. For the next configuration, we need to check if the configuration is better on all instances that have been evaluated for the incumbent. If this is the case, then we can replace the incumbent. Otherwise, a) we need to requeue the config to obtain the missing instance-seed-budget combination or b) mark this configuration as inferior (“rejected”) to not consider it again. The comparison behaviour is controlled by self.get_instance_seed_budget_keys() and self.get_incumbent_instance_seed_budget_keys().

Notably, this method is written to support both multi-fidelity and multi-objective optimization. While the get_instance_seed_budget_keys() method and self.get_incumbent_instance_seed_budget_keys() are used for the multi-fidelity behaviour, calculate_pareto_front() is used as a hard coded way to support multi-objective optimization, including the single objective as special case. calculate_pareto_front() is called on the set of all (in case of MO) incumbents amended with the challenger configuration, provided it has a sufficient overlap in seed-instance-budget combinations.

Lastly, if we have a self._max_incumbents and the pareto front provides more than this specified amount, we cut the incumbents using crowding distance.

Return type:

None

property used_walltime: float

Returns used wallclock time.

abstract property uses_budgets: bool

If the intensifier needs to make use of budgets.

abstract property uses_instances: bool

If the intensifier needs to make use of instances.

abstract property uses_seeds: bool

If the intensifier needs to make use of seeds.

previous

smac.intensifier

next

smac.intensifier.hyperband