smac.runhistory.runhistory

Classes

RunHistory([multi_objective_algorithm, ...])

Container for the target function run information.

Interfaces

class smac.runhistory.runhistory.RunHistory(multi_objective_algorithm=None, overwrite_existing_trials=False)[source]

Bases: Mapping[TrialKey, TrialValue]

Container for the target function run information.

Most importantly, the runhistory contains an efficient mapping from each evaluated configuration to the empirical cost observed on either the full instance set or a subset. The cost is the average over all observed costs for one configuration:

  • If using budgets for a single instance, only the cost on the highest observed budget is returned.

  • If using instances as the budget, the average cost over all evaluated instances is returned.

  • Theoretically, the runhistory object can handle instances and budgets at the same time. This is neither used nor tested.

Note

Guaranteed to be picklable.

Parameters:

  • multi_objective_algorithm (AbstractMultiObjectiveAlgorithm | None, defaults to None) – The multi-objective algorithm is required to scalarize the costs in case of multi-objective.

  • overwrite_existing_trials (bool, defaults to false) – Overwrites a trial (combination of configuration, instance, budget and seed) if it already exists.

__contains__(k)[source]

Dictionary semantics for k in runhistory.

Return type:

bool

__eq__(other)[source]

Enables to check equality of runhistory if the run is continued.

Return type:

bool

__getitem__(k)[source]

Dictionary semantics for v = runhistory[k].

Return type:

TrialValue

__iter__()[source]

Dictionary semantics for for k in runhistory.keys().

Return type:

Iterator[TrialKey]

__len__()[source]

Enables the len(runhistory)

Return type:

int

add(config, cost, time=0.0, status=StatusType.SUCCESS, instance=None, seed=None, budget=None, starttime=0.0, endtime=0.0, additional_info=None, force_update=False)[source]

Adds a new trial to the RunHistory.

Parameters:

  • config (Configuration)

  • cost (int | float | list[int | float]) – Cost of the evaluated trial. Might be a list in case of multi-objective.

  • time (float) – How much time was needed to evaluate the trial.

  • status (StatusType, defaults to StatusType.SUCCESS) – The status of the trial.

  • instance (str | None, defaults to none)

  • seed (int | None, defaults to none)

  • budget (float | None, defaults to none)

  • starttime (float, defaults to 0.0)

  • endtime (float, defaults to 0.0)

  • additional_info (dict[str, Any], defaults to {})

  • force_update (bool, defaults to false) – Overwrites a previous trial if the trial already exists.

Return type:

None

add_running_trial(trial)[source]

Adds a running trial to the runhistory.

Parameters:

trial (TrialInfo) – The TrialInfo object of the running trial.

Return type:

None

add_trial(info, value)[source]

Adds a trial to the runhistory.

Parameters:

trial (TrialInfo) – The TrialInfo object of the running trial.

Return type:

None

average_cost(config, instance_seed_budget_keys=None, normalize=False)[source]

Return the average cost of a configuration. This is the mean of costs of all instance- seed pairs.

Parameters:

  • config (Configuration) – Configuration to calculate objective for.

  • instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.

  • normalize (bool, optional (default=False)) – Normalizes the costs wrt. objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.

Returns:

Cost – Average cost. In case of multiple objectives, the mean of each objective is returned.

Return type:

float | list[float]

property config_ids: dict[Configuration, int]

Mapping from configuration to config id.

empty()[source]

Check whether the RunHistory is empty.

Returns:

emptiness – True if trials have been added to the RunHistory.

Return type:

bool

property finished: int

Returns how many trials have been finished.

get_config(config_id)[source]

Returns the configuration from the configuration id.

Return type:

Configuration

get_config_id(config)[source]

Returns the configuration id from a configuration.

Return type:

int

get_configs(sort_by=None)[source]

Return all configurations in this RunHistory object.

Parameters:

sort_by (str | None, defaults to None) – Sort the configs by cost (lowest cost first) or num_trials (config with lowest number of trials first).

Returns:

configurations – All configurations in the runhistory.

Return type:

list

get_configs_per_budget(budget_subset=None)[source]

Return all configs in this runhistory that have been run on one of these budgets.

Parameters:

budget_subset (list[float | int | None] | None, defaults to None)

Returns:

configurations – List of configurations that have been run on the budgets in budget_subset.

Return type:

list

get_cost(config)[source]

Returns empirical cost for a configuration. See the class docstring for how the costs are computed. The costs are not re-computed, but are read from cache.

Parameters:

config (Configuration)

Returns:

cost – Computed cost for configuration

Return type:

float

get_instance_seed_budget_keys(config, highest_observed_budget_only=True)[source]

Uses get_trials to return a list of instance-seed-budget keys.

Warning

Does not return running instances.

Parameters:

  • config (Configuration)

  • highest_observed_budget_only (bool, defaults to True) – Select only the highest observed budget run for this configuration.

Return type:

list[InstanceSeedBudgetKey]

get_min_cost(config)[source]

Returns the lowest empirical cost for a configuration across all trials.

See the class docstring for how the costs are computed. The costs are not re-computed but are read from cache.

Parameters:

config (Configuration)

Returns:

min_cost – Computed cost for configuration

Return type:

float

get_running_configs()[source]

Returns all configurations which have at least one running trial.

Returns:

List of configurations, all of which have at least one running trial.

Return type:

list[Configuration]

get_running_trials(config=None)[source]

Returns all running trials for the passed configuration.

Parameters:

config (Configuration | None, defaults to None) – Return only running trials from the passed configuration. If None, all configs are considered.

Returns:

trials – List of trials, all of which are still running.

Return type:

list[TrialInfo]

get_trials(config, highest_observed_budget_only=True)[source]

Returns all trials for a configuration.

Warning

Does not return running trials. Please use get_running_trials to receive running trials.

Parameters:

  • config (Configuration)

  • highest_observed_budget_only (bool) – Select only the highest observed budget run for this configuration. Meaning on multiple executions of the same instance-seed pair for a a given configuration, only the highest observed budget is returned.

Returns:

trials – List of trials for the passed configuration.

Return type:

list[InstanceSeedBudgetKey]

has_config(config)[source]

Check if the config is stored in the runhistory

Return type:

bool

property ids_config: dict[int, Configuration]

Mapping from config id to configuration.

incremental_update_cost(config, cost)[source]

Incrementally updates the performance of a configuration by using a moving average.

Parameters:

  • config (Configuration) – configuration to update cost based on all trials in runhistory

  • cost (float) – cost of new run of config

Return type:

None

load(filename, configspace)[source]

Loads the runhistory from disk.

Warning

Overwrites the current runhistory.

Parameters:

  • filename (str | Path)

  • configspace (ConfigSpace)

Return type:

None

min_cost(config, instance_seed_budget_keys=None, normalize=False)[source]
Return the minimum cost of a configuration. This is the minimum cost of all instance-seed

pairs.

Warning

In the case of multi-fidelity, the minimum cost per objectives is returned.

Parameters:

  • config (Configuration) – Configuration to calculate objective for.

  • instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.

  • normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.

Returns:

min_cost – Minimum cost of the config. In case of multi-objective, the minimum cost per objective is returned.

Return type:

float | list[float]

property multi_objective_algorithm: AbstractMultiObjectiveAlgorithm | None

The multi-objective algorithm required to scaralize the costs in case of multi-objective.

property objective_bounds: list[tuple[float, float]]

Returns the lower and upper bound of each objective.

reset()[source]

Resets this runhistory to its default state.

Return type:

None

property running: int

Returns how many trials are still running.

save(filename='runhistory.json')[source]

Saves RunHistory to disk.

Parameters:

filename (str | Path, defaults to "runhistory.json")

Return type:

None

property submitted: int

Returns how many trials have been submitted.

sum_cost(config, instance_seed_budget_keys=None, normalize=False)[source]

Return the sum of costs of a configuration. This is the sum of costs of all instance-seed pairs.

Parameters:

  • config (Configuration) – Configuration to calculate objective for.

  • instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.

  • normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.

Returns:

sum_cost – Sum of costs of config. In case of multiple objectives, the costs are summed up for each objective individually.

Return type:

float | list[float]

update(runhistory)[source]

Updates the current RunHistory by adding new trials from another RunHistory.

Parameters:

runhistory (RunHistory) – RunHistory with additional data to be added to self

Return type:

None

update_cost(config)[source]

Stores the performance of a configuration across the instances in self._cost_per_config and also updates self._num_trials_per_config.

Parameters:

config (Configuration) – configuration to update cost based on all trials in runhistory

Return type:

None

update_costs(instances=None)[source]

Computes the cost of all configurations from scratch and overwrites self._cost_per_config and self._num_trials_per_config accordingly.

Parameters:

instances (list[str] | None, defaults to none) – List of instances; if given, cost is only computed wrt to this instance set.

Return type:

None

update_from_json(filename, configspace)[source]

Updates the current RunHistory by adding new trials from a json file.

Parameters:

  • filename (str) – File name to load from.

  • configspace (ConfigurationSpace)

Return type:

None

previous

smac.runhistory.errors

next

smac.multi_objective