smac.runhistory.runhistory¶
Classes¶
|
Container for the target function run information. |
Interfaces¶
- class smac.runhistory.runhistory.RunHistory(multi_objective_algorithm=None, overwrite_existing_trials=False)[source]¶
Bases:
Mapping
[TrialKey
,TrialValue
]Container for the target function run information.
Most importantly, the runhistory contains an efficient mapping from each evaluated configuration to the empirical cost observed on either the full instance set or a subset. The cost is the average over all observed costs for one configuration:
If using budgets for a single instance, only the cost on the highest observed budget is returned.
If using instances as the budget, the average cost over all evaluated instances is returned.
Theoretically, the runhistory object can handle instances and budgets at the same time. This is neither used nor tested.
Note
Guaranteed to be picklable.
- Parameters:
multi_objective_algorithm (AbstractMultiObjectiveAlgorithm | None, defaults to None) – The multi-objective algorithm is required to scalarize the costs in case of multi-objective.
overwrite_existing_trials (bool, defaults to false) – Overwrites a trial (combination of configuration, instance, budget and seed) if it already exists.
- __eq__(other)[source]¶
Enables to check equality of runhistory if the run is continued.
- Return type:
bool
- __iter__()[source]¶
Dictionary semantics for for k in runhistory.keys().
- Return type:
Iterator
[TrialKey
]
- add(config, cost, time=0.0, status=StatusType.SUCCESS, instance=None, seed=None, budget=None, starttime=0.0, endtime=0.0, additional_info=None, force_update=False)[source]¶
Adds a new trial to the RunHistory.
- Parameters:
config (Configuration)
cost (int | float | list[int | float]) – Cost of the evaluated trial. Might be a list in case of multi-objective.
time (float) – How much time was needed to evaluate the trial.
status (StatusType, defaults to StatusType.SUCCESS) – The status of the trial.
instance (str | None, defaults to none)
seed (int | None, defaults to none)
budget (float | None, defaults to none)
starttime (float, defaults to 0.0)
endtime (float, defaults to 0.0)
additional_info (dict[str, Any], defaults to {})
force_update (bool, defaults to false) – Overwrites a previous trial if the trial already exists.
- Return type:
None
- add_running_trial(trial)[source]¶
Adds a running trial to the runhistory.
- Parameters:
trial (TrialInfo) – The
TrialInfo
object of the running trial.- Return type:
None
- add_trial(info, value)[source]¶
Adds a trial to the runhistory.
- Parameters:
trial (TrialInfo) – The
TrialInfo
object of the running trial.- Return type:
None
- average_cost(config, instance_seed_budget_keys=None, normalize=False)[source]¶
Return the average cost of a configuration. This is the mean of costs of all instance- seed pairs.
- Parameters:
config (Configuration) – Configuration to calculate objective for.
instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.
normalize (bool, optional (default=False)) – Normalizes the costs wrt. objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.
- Returns:
Cost – Average cost. In case of multiple objectives, the mean of each objective is returned.
- Return type:
float | list[float]
- property config_ids: dict[Configuration, int]¶
Mapping from configuration to config id.
- empty()[source]¶
Check whether the RunHistory is empty.
- Returns:
emptiness – True if trials have been added to the RunHistory.
- Return type:
bool
- property finished: int¶
Returns how many trials have been finished.
- get_config(config_id)[source]¶
Returns the configuration from the configuration id.
- Return type:
Configuration
- get_configs(sort_by=None)[source]¶
Return all configurations in this RunHistory object.
- Parameters:
sort_by (str | None, defaults to None) – Sort the configs by
cost
(lowest cost first) ornum_trials
(config with lowest number of trials first).- Returns:
configurations – All configurations in the runhistory.
- Return type:
list
- get_configs_per_budget(budget_subset=None)[source]¶
Return all configs in this runhistory that have been run on one of these budgets.
- Parameters:
budget_subset (list[float | int | None] | None, defaults to None)
- Returns:
configurations – List of configurations that have been run on the budgets in
budget_subset
.- Return type:
list
- get_cost(config)[source]¶
Returns empirical cost for a configuration. See the class docstring for how the costs are computed. The costs are not re-computed, but are read from cache.
- Parameters:
config (Configuration)
- Returns:
cost – Computed cost for configuration
- Return type:
float
- get_instance_seed_budget_keys(config, highest_observed_budget_only=True)[source]¶
Uses
get_trials
to return a list of instance-seed-budget keys.Warning
Does not return running instances.
- Parameters:
config (Configuration)
highest_observed_budget_only (bool, defaults to True) – Select only the highest observed budget run for this configuration.
- Return type:
list[InstanceSeedBudgetKey]
- get_min_cost(config)[source]¶
Returns the lowest empirical cost for a configuration across all trials.
See the class docstring for how the costs are computed. The costs are not re-computed but are read from cache.
- Parameters:
config (Configuration)
- Returns:
min_cost – Computed cost for configuration
- Return type:
float
- get_running_configs()[source]¶
Returns all configurations which have at least one running trial.
- Returns:
List of configurations, all of which have at least one running trial.
- Return type:
list[Configuration]
- get_running_trials(config=None)[source]¶
Returns all running trials for the passed configuration.
- Parameters:
config (Configuration | None, defaults to None) – Return only running trials from the passed configuration. If None, all configs are considered.
- Returns:
trials – List of trials, all of which are still running.
- Return type:
list[TrialInfo]
- get_trials(config, highest_observed_budget_only=True)[source]¶
Returns all trials for a configuration.
Warning
Does not return running trials. Please use
get_running_trials
to receive running trials.- Parameters:
config (Configuration)
highest_observed_budget_only (bool) – Select only the highest observed budget run for this configuration. Meaning on multiple executions of the same instance-seed pair for a a given configuration, only the highest observed budget is returned.
- Returns:
trials – List of trials for the passed configuration.
- Return type:
list[InstanceSeedBudgetKey]
- property ids_config: dict[int, Configuration]¶
Mapping from config id to configuration.
- incremental_update_cost(config, cost)[source]¶
Incrementally updates the performance of a configuration by using a moving average.
- Parameters:
config (Configuration) – configuration to update cost based on all trials in runhistory
cost (float) – cost of new run of config
- Return type:
None
- load(filename, configspace)[source]¶
Loads the runhistory from disk.
Warning
Overwrites the current runhistory.
- Parameters:
filename (str | Path)
configspace (ConfigSpace)
- Return type:
None
- min_cost(config, instance_seed_budget_keys=None, normalize=False)[source]¶
- Return the minimum cost of a configuration. This is the minimum cost of all instance-seed
pairs.
Warning
In the case of multi-fidelity, the minimum cost per objectives is returned.
- Parameters:
config (Configuration) – Configuration to calculate objective for.
instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.
normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.
- Returns:
min_cost – Minimum cost of the config. In case of multi-objective, the minimum cost per objective is returned.
- Return type:
float | list[float]
- property multi_objective_algorithm: AbstractMultiObjectiveAlgorithm | None¶
The multi-objective algorithm required to scaralize the costs in case of multi-objective.
- property objective_bounds: list[tuple[float, float]]¶
Returns the lower and upper bound of each objective.
- property running: int¶
Returns how many trials are still running.
- save(filename='runhistory.json')[source]¶
Saves RunHistory to disk.
- Parameters:
filename (str | Path, defaults to "runhistory.json")
- Return type:
None
- property submitted: int¶
Returns how many trials have been submitted.
- sum_cost(config, instance_seed_budget_keys=None, normalize=False)[source]¶
Return the sum of costs of a configuration. This is the sum of costs of all instance-seed pairs.
- Parameters:
config (Configuration) – Configuration to calculate objective for.
instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the runhistory is queried for all trials of the given configuration.
normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing. Also, the objective weights are incorporated.
- Returns:
sum_cost – Sum of costs of config. In case of multiple objectives, the costs are summed up for each objective individually.
- Return type:
float | list[float]
- update(runhistory)[source]¶
Updates the current RunHistory by adding new trials from another RunHistory.
- Parameters:
runhistory (RunHistory) – RunHistory with additional data to be added to self
- Return type:
None
- update_cost(config)[source]¶
Stores the performance of a configuration across the instances in self._cost_per_config and also updates self._num_trials_per_config.
- Parameters:
config (Configuration) – configuration to update cost based on all trials in runhistory
- Return type:
None
- update_costs(instances=None)[source]¶
Computes the cost of all configurations from scratch and overwrites self._cost_per_config and self._num_trials_per_config accordingly.
- Parameters:
instances (list[str] | None, defaults to none) – List of instances; if given, cost is only computed wrt to this instance set.
- Return type:
None