smac.runhistory.runhistory¶

Classes

`DataOrigin`(value)	Definition of how data in the runhistory is used.
`EnumEncoder`(*[, skipkeys, ensure_ascii, ...])	Custom encoder for enum-serialization (implemented for StatusType from tae).
`InstSeedBudgetKey`(instance, seed, budget)
`InstSeedKey`(instance, seed)
`RunHistory`([overwrite_existing_runs])	Container for target algorithm run information.
`RunInfo`(config, instance, instance_specific, ...)
`RunKey`(config_id, instance_id, seed[, budget])
`RunValue`(cost, time, status, starttime, ...)

class smac.runhistory.runhistory.DataOrigin(value)[source]¶

Bases: enum.Enum

Definition of how data in the runhistory is used.

INTERNAL: internal data which was gathered during the current optimization run. It will be saved to disk, used for building EPMs and during intensify.
EXTERNAL_SAME_INSTANCES: external data, which was gathered by running
another program on the same instances as the current optimization run runs on (for example pSMAC). It will not be saved to disk, but used both for EPM building and during intensify.
EXTERNAL_DIFFERENT_INSTANCES: external data, which was gathered on a
different instance set as the one currently used, but due to having the same instance features can still provide useful information. Will not be saved to disk and only used for EPM building.

class smac.runhistory.runhistory.EnumEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]¶

Bases: json.encoder.JSONEncoder

Custom encoder for enum-serialization (implemented for StatusType from tae).

Using encoder implied using object_hook as defined in StatusType to deserialize from json.

default(obj)[source]¶

Returns the default encoding of the passed object.

Return type: Any

class smac.runhistory.runhistory.InstSeedBudgetKey(instance, seed, budget)¶

Bases: tuple

budget¶: Alias for field number 2

instance¶: Alias for field number 0

seed¶: Alias for field number 1

class smac.runhistory.runhistory.InstSeedKey(instance, seed)¶

Bases: tuple

instance¶: Alias for field number 0

seed¶: Alias for field number 1

class smac.runhistory.runhistory.RunHistory(overwrite_existing_runs=False)[source]¶

Bases: Mapping[smac.runhistory.runhistory.RunKey, smac.runhistory.runhistory.RunValue]

Container for target algorithm run information.

Most importantly, the runhistory contains an efficient mapping from each evaluated configuration to the empirical cost observed on either the full instance set or a subset. The cost is the average over all observed costs for one configuration:

If using budgets for a single instance, only the cost on the highest observed budget is returned.
If using instances as the budget, the average cost over all evaluated instances is returned.
Theoretically, the runhistory object can handle instances and budgets at the same time. This is neither used nor tested.
Capped runs are not included in this cost.

Note

Guaranteed to be picklable.

Parameters: overwrite_existing_runs (bool (default=False)) – If set to True and a run of a configuration on an instance-budget-seed-pair already exists, it is overwritten. Allows to overwrites old results if pairs of algorithm-instance-seed were measured multiple times

data¶

Internal data representation

Type: collections.OrderedDict()

config_ids¶

Maps config -> id

Type: dict

ids_config¶

Maps id -> config

Type: dict

num_runs_per_config¶

Maps config_id -> number of runs

Type: dict

__contains__(k)[source]¶

Dictionary semantics for k in runhistory

Return type: bool

__getitem__(k)[source]¶

Dictionary semantics for v = runhistory[k]

Return type: RunValue

__iter__()[source]¶

Dictionary semantics for for k in runhistory.keys(), enables.

.items()

Return type: Iterator[RunKey]

__len__()[source]¶

Enables the len(runhistory)

Return type: int

add(config, cost, time, status, instance_id=None, seed=None, budget=0.0, starttime=0.0, endtime=0.0, additional_info=None, origin=DataOrigin.INTERNAL, force_update=False)[source]¶

Adds a data of a new target algorithm (TA) run; it will update data if the same key values are used (config, instance_id, seed)

Parameters

config (dict (or other type -- depending on config space module)) – Parameter configuration
cost (Union[int, float, list, np.ndarray]) – Cost of TA run (will be minimized)
time (float) – Runtime of TA run
status (str) – Status in {SUCCESS, TIMEOUT, CRASHED, ABORT, MEMOUT}
instance_id (str) – String representing an instance (default: None)
seed (int) – Random seed used by TA (default: None)
budget (float) – budget (cutoff) used in intensifier to limit TA (default: 0)
starttime (float) – starting timestamp of TA evaluation
endtime (float) – ending timestamp of TA evaluation
additional_info (dict) – Additional run infos (could include further returned information from TA or fields such as start time and host_id)
origin (DataOrigin) – Defines how data will be used.
force_update (bool (default: False)) – Forces the addition of a config to the history

Return type

None

average_cost(config, instance_seed_budget_keys=None, normalize=False)[source]¶

Return the average cost of a configuration. This is the mean of costs of all instance- seed pairs.

Parameters

config (Configuration) – Configuration to calculate objective for.
instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the run_history is queried for all runs of the given configuration.
normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing.

Returns

Cost – Average cost. In case of multiple objectives, the mean of each objective is returned.

Return type

float | list[float]

compute_all_costs(instances=None)[source]¶

Computes the cost of all configurations from scratch and overwrites self.cost_perf_config and self.runs_per_config accordingly.

Note

This method is only used for merge_foreign_data and should be removed.

Parameters: instances (List[str]) – List of instances; if given, cost is only computed wrt to this instance set.
Return type: None

empty()[source]¶

Check whether or not the RunHistory is empty.

Returns: emptiness – True if runs have been added to the RunHistory, False otherwise
Return type: bool

get_all_configs()[source]¶

Return all configurations in this RunHistory object.

Returns: parameter configurations
Return type: list

get_all_configs_per_budget(budget_subset=None)[source]¶

Return all configs in this RunHistory object that have been run on one of these budgets.

Parameters: budget_subset (list) –
Returns: parameter configurations
Return type: list

get_cost(config)[source]¶

Returns empirical cost for a configuration. See the class docstring for how the costs are computed. The costs are not re-computed, but are read from cache.

Parameters: config (Configuration) –
Returns: cost – Computed cost for configuration
Return type: float

get_instance_costs_for_config(config)[source]¶

Returns the average cost per instance (across seeds) for a configuration. If the runhistory contains budgets, only the highest budget for a configuration is returned.

Note

This is used by the pSMAC facade to determine the incumbent after the evaluation.

Parameters: config (Configuration from ConfigSpace) – Parameter configuration
Returns: cost_per_inst
Return type: Dict<instance name<str>, cost<float>>

get_min_cost(config)[source]¶

Returns the lowest empirical cost for a configuration, across all runs (budgets)

See the class docstring for how the costs are computed. The costs are not re-computed, but are read from cache.

Parameters: config (Configuration) –
Returns: min_cost – Computed cost for configuration
Return type: float

get_runs_for_config(config, only_max_observed_budget)[source]¶

Return all runs (instance seed pairs) for a configuration.

Note

This method ignores capped runs.

Parameters

config (Configuration from ConfigSpace) – Parameter configuration
only_max_observed_budget (bool) – Select only the maximally observed budget run for this configuration

Returns

instance_seed_budget_pairs

Return type

list<tuples of instance, seed, budget>

incremental_update_cost(config, cost)[source]¶

Incrementally updates the performance of a configuration by using a moving average.

Parameters

config (Configuration) – configuration to update cost based on all runs in runhistory
cost (float) – cost of new run of config

Return type

None

load_json(fn, cs)[source]¶

Load and runhistory in json representation from disk.

Warning

Overwrites current runhistory!

Parameters

fn (str) – file name to load from
cs (ConfigSpace) – instance of configuration space

Return type

None

min_cost(config, instance_seed_budget_keys=None, normalize=False)[source]¶

Return the minimum cost of a configuration.

This is the minimum cost of all instance-seed pairs.

Warning

In the case of multi-fidelity, the minimum cost per objectives is returned.

Parameters

config (Configuration) – Configuration to calculate objective for.
instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the run_history is queried for all runs of the given configuration.

Returns

min_cost – Minimum cost of the config. In case of multi-objective, the minimum cost per objective is returned.

Return type

float | list[float]

save_json(fn='runhistory.json', save_external=False)[source]¶

Saves runhistory on disk.

Parameters

fn (str) – file name.
save_external (bool) – Whether to save external data in the runhistory file.

Return type

None

sum_cost(config, instance_seed_budget_keys=None, normalize=False)[source]¶

Return the sum of costs of a configuration. This is the sum of costs of all instance-seed pairs.

Parameters

config (Configuration) – Configuration to calculate objective for.
instance_seed_budget_keys (list, optional (default=None)) – List of tuples of instance-seeds-budget keys. If None, the run_history is queried for all runs of the given configuration.
normalize (bool, optional (default=False)) – Normalizes the costs wrt objective bounds in the multi-objective setting. Only a float is returned if normalize is True. Warning: The value can change over time because the objective bounds are changing.

Returns

sum_cost – Sum of costs of config. In case of multiple objectives, the costs are summed up for each objective individually.

Return type

float | list[float]

update(runhistory, origin=DataOrigin.EXTERNAL_SAME_INSTANCES)[source]¶

Updates the current runhistory by adding new runs from a RunHistory.

Parameters

runhistory (RunHistory) – Runhistory with additional data to be added to self
origin (DataOrigin) – If set to INTERNAL or EXTERNAL_FULL the data will be added to the internal data structure self._configid_to_inst_seed_budget and be available through get_runs_for_config().

Return type

None

update_cost(config)[source]¶

Stores the performance of a configuration across the instances in self.cost_per_config and also updates self.runs_per_config;

Note

This method ignores capped runs.

Parameters: config (Configuration) – configuration to update cost based on all runs in runhistory
Return type: None

update_from_json(fn, cs, origin=DataOrigin.EXTERNAL_SAME_INSTANCES)[source]¶

Updates the current runhistory by adding new runs from a json file.

Parameters

fn (str) – File name to load from.
cs (ConfigSpace) – Instance of configuration space.
origin (DataOrigin) – What to store as data origin.

Return type

None

class smac.runhistory.runhistory.RunInfo(config: ConfigSpace.configuration_space.Configuration, instance: Optional[str], instance_specific: str, seed: int, cutoff: Optional[float], capped: bool, budget: float = 0.0, source_id: int = 0)[source]¶

Bases: smac.runhistory.runhistory.RunInfo

static __new__(cls, config, instance, instance_specific, seed, cutoff, capped, budget=0.0, source_id=0)[source]¶

Creates a new RunInfo instance.

Return type: RunInfo

class smac.runhistory.runhistory.RunKey(config_id: int, instance_id: Optional[str], seed: Optional[int], budget: float = 0.0)[source]¶

Bases: smac.runhistory.runhistory.RunKey

static __new__(cls, config_id, instance_id, seed, budget=0.0)[source]¶

Creates a new RunKey instance.

Return type: RunKey

class smac.runhistory.runhistory.RunValue(cost, time, status, starttime, endtime, additional_info)¶

Bases: tuple

additional_info¶: Alias for field number 5

cost¶: Alias for field number 0

endtime¶: Alias for field number 4

starttime¶: Alias for field number 3

status¶: Alias for field number 2

time¶: Alias for field number 1

smac.runhistory

smac.runhistory.runhistory2epm

SMAC3 Documentation

smac.runhistory.runhistory¶