smac.runhistory.runhistory

Classes

RunHistory([multi_objective_algorithm, ...])

Container for the target function run information.

Interfaces

class smac.runhistory.runhistory.RunHistory(multi_objective_algorithm=None, overwrite_existing_trials=False)[source]

Bases: Mapping[TrialKey, TrialValue]

Container for the target function run information.

Most importantly, the runhistory contains an efficient mapping from each evaluated configuration to the empirical cost observed on either the full instance set or a subset. The cost is the average over all observed costs for one configuration:

  • If using budgets for a single instance, only the cost on the highest observed budget is returned.

  • If using instances as the budget, the average cost over all evaluated instances is returned.

  • Theoretically, the runhistory object can handle instances and budgets at the same time. This is neither used nor tested.

Note

Guaranteed to be picklable.

Parameters:

  • multi_objective_algorithm (AbstractMultiObjectiveAlgorithm | None, defaults to None) – The multi-objective algorithm is required to scaralize the costs in case of multi-objective.

  • overwrite_existing_trials (bool, defaults to false) – Overwrites a trial (combination of configuration, instance, budget and seed) if it already exists.

__contains__(k)[source]

Dictionary semantics for k in runhistory.

Return type:

bool

__eq__(other)[source]

Enables to check equality of runhistory if the run is continued.

Return type:

bool

__getitem__(k)[source]

Dictionary semantics for v = runhistory[k].

Return type:

TrialValue

__iter__()[source]

Dictionary semantics for for k in runhistory.keys().

Return type:

Iterator[TrialKey]

__len__()[source]

Enables the len(runhistory)

Return type:

int

add(config, cost, time, status=StatusType.SUCCESS, instance=None, seed=None, budget=None, starttime=0.0, endtime=0.0, additional_info={}, origin=DataOrigin.INTERNAL, force_update=False)[source]

Adds a new trial.

Parameters:

  • config (Configuration) –

  • cost (int | float | list[int | float]) – Cost of the evaluated trial. Might be a list in case of multi-objective.

  • time (float) – How much time was needed to evaluate the trial.

  • status (StatusType, defaults to StatusType.SUCCESS) – The status of the trial.

  • instance (str | None, defaults to none) –

  • seed (int | None, defaults to none) –

  • budget (float | None, defaults to none) –

  • starttime (float, defaults to 0.0) –

  • endtime (float, defaults to 0.0) –

  • additional_info (dict[str, Any], defaults to {}) –

  • origin (DataOrigin, defaults to DataOrigin.INTERNAL) –

  • force_update (bool, defaults to false) – Overwrites a previous trial if the trial already exists.

Return type:

None

average_cost(config, instance_seed_budget_keys=None, normalize=False)[source]

Return the average cost of a configuration. This is the mean of costs of all instance- seed pairs.

Parameters:

  • config (Configuration) – Configuration to calculate objective for.

  • instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.

  • normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.

Returns:

Cost – Average cost. In case of multiple objectives, the mean of each objective is returned.

Return type:

float | list[float]

property config_ids: dict[ConfigSpace.configuration_space.Configuration, int]

Mapping from configuration to config id.

Return type:

dict[Configuration, int]

empty()[source]

Check whether or not the RunHistory is empty.

Returns:

emptiness – True if trials have been added to the RunHistory.

Return type:

bool

get_config(config_id)[source]

Returns the configuration from the configuration id.

Return type:

Configuration

get_configs()[source]

Return all configurations in this RunHistory object.

Returns:

parameter configurations

Return type:

list

get_configs_per_budget(budget_subset=None)[source]

Return all configs in this RunHistory object that have been run on one of these budgets.

Parameters:

budget_subset (list) –

Returns:

parameter configurations

Return type:

list

get_cost(config)[source]

Returns empirical cost for a configuration. See the class docstring for how the costs are computed. The costs are not re-computed, but are read from cache.

Parameters:

config (Configuration) –

Returns:

cost – Computed cost for configuration

Return type:

float

get_incumbent()[source]

Returns the incumbent configuration. The config with the lowest cost calculated by get_cost is returned.

Warning

The incumbent in a multi-objective setting depends on the multi-objective algorithm. If you use ParEGO, for example, you get a random incumbent on the Pareto front based on the current ParEGO weights.

Return type:

tuple[Optional[Configuration], float | list[float]]

get_min_cost(config)[source]

Returns the lowest empirical cost for a configuration across all trials.

See the class docstring for how the costs are computed. The costs are not re-computed but are read from cache.

Parameters:

config (Configuration) –

Returns:

min_cost – Computed cost for configuration

Return type:

float

get_pareto_front()[source]

Returns the Pareto front of the runhistory.

Return type:

tuple[list[Configuration], list[list[float]]]

Returns:

  • configs (list[Configuration]) – The configs of the Pareto front.

  • costs (list[list[float]]) – The costs from the configs of the Pareto front.

get_trials(config, only_max_observed_budget=True)[source]

Return all trials (instance seed budget key in this case) for a configuration.

Parameters:

  • config (Configuration) – Parameter configuration

  • only_max_observed_budget (bool) – Select only the maximally observed budget run for this configuration

Returns:

trials

Return type:

list[InstanceSeedBudgetKey]

property ids_config: dict[int, ConfigSpace.configuration_space.Configuration]

Mapping from config id to configuration.

Return type:

dict[int, Configuration]

incremental_update_cost(config, cost)[source]

Incrementally updates the performance of a configuration by using a moving average.

Parameters:

  • config (Configuration) – configuration to update cost based on all trials in runhistory

  • cost (float) – cost of new run of config

Return type:

None

load_json(filename, configspace)[source]

Load and runhistory in json representation from disk.

Warning

Overwrites current runhistory!

Parameters:

  • filename (str) – file name to load from

  • configspace (ConfigSpace) – instance of configuration space

Return type:

None

min_cost(config, instance_seed_budget_keys=None, normalize=False)[source]

Return the minimum cost of a configuration. This is the minimum cost of all instance-seed pairs.

Warning

In the case of multi-fidelity, the minimum cost per objectives is returned.

Parameters:

  • config (Configuration) – Configuration to calculate objective for.

  • instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.

  • normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.

Returns:

min_cost – Minimum cost of the config. In case of multi-objective, the minimum cost per objective is returned.

Return type:

float | list[float]

property multi_objective_algorithm: smac.multi_objective.abstract_multi_objective_algorithm.AbstractMultiObjectiveAlgorithm | None

The multi-objective algorithm is required to scaralize the costs in case of multi-objective.

Return type:

Optional[AbstractMultiObjectiveAlgorithm]

property objective_bounds: list[tuple[float, float]]

Returns the lower and upper bound of each objective.

Return type:

list[tuple[float, float]]

reset()[source]

Resets this runhistory to it’s default state.

Return type:

None

save_json(filename='runhistory.json', save_external=False)[source]

Saves runhistory on disk.

Parameters:

  • filename (str) – file name.

  • save_external (bool) – Whether to save external data in the runhistory file.

Return type:

None

sum_cost(config, instance_seed_budget_keys=None, normalize=False)[source]

Return the sum of costs of a configuration. This is the sum of costs of all instance-seed pairs.

Parameters:

  • config (Configuration) – Configuration to calculate objective for.

  • instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.

  • normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.

Returns:

sum_cost – Sum of costs of config. In case of multiple objectives, the costs are summed up for each objective individually.

Return type:

float | list[float]

update(runhistory, origin=DataOrigin.EXTERNAL_SAME_INSTANCES)[source]

Updates the current runhistory by adding new trials from a RunHistory.

Parameters:

  • runhistory (RunHistory) – Runhistory with additional data to be added to self

  • origin (DataOrigin, defaults to DataOrigin.EXTERNAL_SAME_INSTANCES) – If set to INTERNAL or EXTERNAL_FULL the data will be added to the internal data structure self._config_id_to_inst_seed_budget and be available through get_trials().

Return type:

None

update_cost(config)[source]

Stores the performance of a configuration across the instances in self._cost_per_config and also updates self._num_trials_per_config.

Parameters:

config (Configuration) – configuration to update cost based on all trials in runhistory

Return type:

None

update_costs(instances=None)[source]

Computes the cost of all configurations from scratch and overwrites self._cost_per_config and self._num_trials_per_config accordingly.

Parameters:

instances (list[str] | None, defaults to none) – List of instances; if given, cost is only computed wrt to this instance set.

Return type:

None

update_from_json(filename, configspace, origin=DataOrigin.EXTERNAL_SAME_INSTANCES)[source]

Updates the current runhistory by adding new trials from a json file.

Parameters:

  • filename (str) – File name to load from.

  • configspace (ConfigurationSpace) –

  • origin (DataOrigin, defaults to DataOrigin.EXTERNAL_SAME_INSTANCES) – What to store as data origin.

Return type:

None

previous

smac.runhistory.enumerations

next

smac.multi_objective