deepcave.runs

# AbstractRun

This module provides utilities to create and handle an abstract run.

It provides functions to get information of the run, as well as the used objectives.

## Classes

• AbstractRun: Create a new run.

Functions

check_equality(runs[, meta, configspace, ...])

Check the passed runs on equality based on the selected runs.

Classes

AbstractRun(name)

Create a new run.

class deepcave.runs.AbstractRun(name)

Bases: ABC

Create a new run.

Provide functions to get information of the run, as well as the used objectives.

Properties

namestr

The name of the run.

pathOptional[Path]

The path to the run.

loggerLogger

The logger for the run.

meta: Dict[str, Any]

Contains the run’s meta information.

configspace: ConfigSpace.ConfigurationSpace

The configuration space of the run.

configs: Dict[int, Configuration]

Contains the configurations.

origins: Dict[int, str]

The origin of the configuration.

models: Dict[int, Optional[Union[str, “torch.nn.Module”]]]

Contains the modules.

history: List[Trial]

The history of Trials.

trial_keys: Dict[Tuple[str, int], int]

Contains config_id, budget and the corresponding trial_id.

models_dirPath

The directory of the model.

empty()

Check if the run’s history is empty.

Returns:

True if run history is empty, False otherwise.

Return type:

bool

encode_config(config, specific=False)

Encode a given configuration (id) to a normalized list.

If a configuration is passed, no look-up has to be done.

Parameters:

• config (Union[int, Dict[Any, Any], Configuration]) – Either the configuration id, as configuration as dict, or a Configuration itself.

• specific (bool) – Use specific encoding for fanova tree, by default False.

Returns:

The encoded config as list.

Return type:

List

encode_configs(configs)

Encode a list of configurations into a corresponding numpy array.

Parameters:

configs (List[Configuration]) – A list containing the configurations to be encoded.

Returns:

A numpy array with the encoded configurations.

Return type:

np.ndarray

get_all_costs(budget=None, statuses=None, seed=None, selected_ids=None)

Get all costs in the history with their config ids and seeds.

Optionally, only configurations which were evaluated on the passed budget, seed, and stati are considered.

In case of multi-objective, multiple costs are returned in the form of a list.

Parameters:

• budget (Optional[Union[int, float]]) – Budget to select the costs. If no budget is given, the highest budget is chosen. By default None.

• statuses (Optional[Union[Status, List[Status]]]) – Only selected stati are considered. If no status is given, all stati are considered. By default None.

• seed (Optional[int], optional) – Seed to select the costs. If no seed is given, all seeds are considered. By default None.

• selected_ids (Optional[List[int]], optional) – If set, only history ids in the list will be considered. By default None.

Returns:

Config ids and seeds with their corresponding list of costs.

Return type:

Dict[int, Dict[int, List[float]]]

get_avg_costs(config_id, budget=None, statuses=None)

Get average costs over all seeds for a config.

Optionally, only configurations which were evaluated on the passed budget and stati are considered.

In case of multi-objective, multiple costs are returned in the form of a list.

Parameters:

• config_id (int) – Configuration id to get the costs for.

• budget (Optional[Union[int, float]]) – Budget to get the costs from the configuration id for. If budget is None, the highest budget is chosen. By default None.

• statuses (Optional[Union[Status, List[Status]]]) – Only selected stati are considered. If no status is given, all stati are considered. By default None.

Return type:

Tuple[List[float], List[float]]

Returns:

• List[float] – List of average cost values for the given config_id and budget.

• List[float] – List of std cost values for the given config_id and budget.

get_budget(id, human=False)

Get the budget given an id.

Parameters:

• id (Union[int, str]) – The id of the wanted budget. If id is a string, it is converted to an integer.

• human (bool, optional) – Make the output more readable. By default False.

Returns:

The budget.

Return type:

float, int

Raises:

TypeError – If the budget with this id is invalid.

get_budget_ids(include_combined=True)

Get the corresponding ids for the budgets.

Parameters:

include_combined (bool, optional) – If False, cut last id of budget ids. By default True.

Returns:

A list of the budget ids.

Return type:

List[int]

get_budgets(human=False, include_combined=True)

Return the budgets from the meta data.

Parameters:

• human (bool, optional) – Make the output more readable. By default False.

• include_combined (bool, optional) – If True, include the combined budget. By default True.

Returns:

List of budgets. In a readable form, if human is True.

Return type:

List[Union[int, float, str]]

get_config(id)

Retrieve the configuration with the corresponding id.

Parameters:

id (int) – The id of the configuration.

Returns:

The corresponding Configuration.

Return type:

Configuration

get_config_id(config)

Get the id of the configuration.

Parameters:

config (Union[Configuration, Dict]) – The configuration for which to find the id.

Returns:

The configuration id.

Return type:

Optional[int]

get_configs(budget=None, seed=None, statuses=None)

Get configurations of the run.

Optionally, only configurations which were evaluated on the passed budget, seed, and stati are considered.

Parameters:

• budget (Optional[Union[int, float]]) – Budget to select the configs. If no budget is given, all seeds are considered. By default None.

• seed (Optional[int]) – Seed to select the configs. If no seed is given, all seeds are considered. By default None.

• statuses (Optional[Union[Status, List[Status]]]) – Only selected stati are considered. If no status is given, all stati are considered. By default None.

Returns:

Configuration id and the configuration.

Return type:

Dict[int, Configuration]

get_encoded_data(objectives=None, budget=None, seed=None, statuses=None, specific=False, include_config_ids=False, include_combined_cost=False)

Encode configurations to process them further.

After the configurations are encoded, they can be used in model prediction.

Parameters:

• objectives (Optional[Union[Objective, List[Objective]]]) – Which objectives should be considered. If None, all objectives are considered. By default None.

• budget (Optional[List[Status]]) – Which budget should be considered. If None, only the highest budget is considered. By default None.

• seed (Optional[int]) – Which seed should be considered. If None, all seeds are considered. By default None.

• statuses (Optional[Union[Status, List[Status]]]) – Which stati should be considered. If None, all stati are considered. By default None.

• specific (bool) – Whether a specific encoding should be used. This encoding is compatible with pyrfr. A wrapper for pyrfr is implemented in deepcave.evaluators.epm. By default False.

• include_config_ids (bool) – Whether to include configuration ids. By default False.

• include_combined_cost (bool, optional) – Whether to include combined cost. Note that the combined cost is calculated by the passed objectives only. By default False.

Returns:

df – Encoded dataframe with the following columns (depending on the parameters): [CONFIG_ID, HP1, HP2, …, HPn, OBJ1, OBJ2, …, OBJm, COMBINED_COST]

Return type:

pd.DataFrame

Raises:

ValueError – If a hyperparameter (HP) is not supported.

get_highest_budget(config_id=None)

Return the highest found budget for a config id.

If no config id is specified then the highest available budget is returned. Moreover, if no budget is available None is returned.

Parameters:

config_id (Optional[int]) – The config id for which the highest budget is returned.

Returns:

The highest budget or None if no budget was specified.

Return type:

Optional[Union[int, float]]

get_incumbent(objectives=None, budget=None, seed=None, statuses=None, selected_ids=None)

Return the incumbent with its normalized objective value.

The incumbent is the configuration with the lowest normalized objective value.

Optionally, only configurations which were evaluated on the passed budget, seed, and stati are considered.

Parameters:

• objectives (Optional[Union[Objective, List[Objective]]], optional) – Considered objectives. If None, all objectives are considered. By default None.

• budget (Optional[Union[int, float]], optional) – Considered budget. If None, the highest budget is chosen. By default None.

• seed (Optional[int], optional) – Considered seed. If no seed is given, all seeds are considered. By default None.

• statuses (Optional[Union[Status, List[Status]]], optional) – Considered stati. If None, all stati are considered. By default None.

• selected_ids (Optional[List[int]], optional) – If set, only ids in selected_ids will be considered. This can for example be useful if only ids up to a certain end-time shall be considered. By default None.

Returns:

Incumbent with its normalized cost.

Return type:

Tuple[Configuration, float]

Raises:

RuntimeError – If no incumbent was found.

get_meta()

Get a shallow copy of the meta information.

Returns:

A shallow copy of the meta information dictionary.

Return type:

Dict[str, Any]

get_model(config_id)

Get a model associated with the configuration id.

Parameters:

config_id (int) – The configuration id.

Returns:

A model for the provided configuration id.

Return type:

Optional[“torch.nn.Module]

get_num_configs(budget=None, seed=None)

Count the number of configurations stored in this run with a specific budget.

Parameters:

• budget (Optional[Union[int, float]]) – The budget for which to count the configurations. If not provided, counts all configurations. Default is None.

• seed (Optional[int]) – The seed for which to count the configurations. If not provided, counts all configurations. Default is None.

Returns:

The number of all configurations with a given budget. If budget is None, counts all configurations.

Return type:

int

get_objective(id)

Return the objective based on the id or the name.

Parameters:

id (Union[str, int]) – The id or name of the objective.

Returns:

The objective.

Return type:

Optional[Objective]

get_objective_id(objective)

Return the id of the objective if it is found.

Parameters:

objective (Union[Objective, str]) – The objective or objective name for which the id is returned.

Returns:

objective_id – Objective id from the passed objective.

Return type:

int

Raises:

RuntimeError – If objective was not found.

get_objective_ids()

Get the ids of the objectives.

Returns:

A list of the ids of the objectives.

Return type:

List[int]

get_objective_name(objectives=None)

Get the cost name of given objective names.

Returns “Combined Cost” if multiple objective names were involved.

Parameters:

objectives (Optional[List[Objective]]) – A list of the objectives. By default None.

Returns:

The name of the objective. Returns “Combined Cost” if multiple objective names were involved.

Return type:

str

get_objective_names()

Get the names of the objectives.

Returns:

A list containing the names of the objectives.

Return type:

List[str]

get_objectives()

Get a list of all objectives corresponding to the run.

Returns:

A list containing all objectives associated with the run.

Return type:

List[Objective]

get_origin(config_id)

Get the origin, given a config id.

Parameters:

config_id (int) – The identificator of the configuration.

Returns:

An origin string corresponding to the given configuration id.

Return type:

Optional[str]

get_seeds(human=False, include_combined=True)

Return the seeds from the meta data.

Parameters:

• human (bool, optional) – Make the output better readable. By default False.

• include_combined (bool, optional) – If true, return combined seed as well. By default True.

Returns:

List of seeds.

Return type:

List[Union[int, str]]

get_status(config_id, seed, budget=None)

Return the status of a trial (i.e. configuration, budget and seed).

Parameters:

• config_id (int) – Configuration id to get the status for.

• seed (Optional[int]) – Seed to get the status from the configuration id for.

• budget (Optional[Union[int, float]]) – Budget to get the status from the configuration id for. If budget is None, the highest budget is chosen. By default None.

Returns:

Status of the configuration.

Return type:

Status

Raises:

ValueError – If the configuration id is not found.

get_trajectory(objective, budget=None, seed=None)

Calculate the trajectory of the given objective, budget, and seed.

Parameters:

• objective (Objective) – Objective to calculate the trajectory for.

• budget (Optional[Union[int, float]]) – Budget to calculate the trajectory for. If no budget is given, then the highest budget is chosen. By default None.

• seed (Optional[int], optional) – Seed to calculate the trajectory for. If no seed is given, then all seeds are considered. By default None.

Returns:

timesList[float]

Times of the trajectory.

costs_meanList[float]

Costs of the trajectory.

costs_stdList[float]

Standard deviation of the costs of the trajectory. This is particularly useful for grouped runs.

idsList[int]

The “global” ids of the selected trials.

config_idsList[int]

Config ids of the selected trials.

Return type:

Tuple[List[float], List[float], List[float], List[int], List[int]]

get_trial(trial_key)

Get the trial with the responding key if existing.

Parameters:

trial_key (Tuple[int, Union[int, float], int]) – The key for the desired trial.

Returns:

The trial object.

Return type:

Optional[Trial]

static get_trial_key(config_id, budget, seed)

Get the trial key.

It is obtained through the assembly of configuration and budget.

Parameters:

• config_id (int) – The identificator of the configuration.

• budget (Optional[Union[int, float]]) – The budget of the Trial.

• seed (Optional[int]) – The seed used for the Trial.

Returns:

Tuple representing the trial key, consisting of configuration id, budget, and seed.

Return type:

Tuple[int, Optional[Union[int, float]], Optional[int]]

get_trials()

Get an iterator of all stored trials.

Returns:

An iterator over all stored trials.

Return type:

Iterator[Trial]

abstract property hash: str

Hash of the current run.

If hash changes, cache has to be cleared. This ensures that the cache always holds the latest results of the run.

Returns:

hash – Hash of the run.

Return type:

str

abstract property id: str

Hash of the file.

This is used to identify the file. In contrast to hash, this hash should not be changed throughout the run.

Returns:

Hash of the run.

Return type:

str

property latest_change: float

Get the latest change.

Returns:

The latest change.

Return type:

float

merge_costs(costs, objectives=None)

Calculate one cost value from multiple costs.

Normalizes the costs first and weigh every cost the same. The lower the normalized cost, the better.

Parameters:

• costs (List[float]) – The costs, which should be merged. Must be the same length as the original number of objectives.

• objectives (Optional[List[Objective]]) – The considered objectives to the costs. By default None. If None, all objectives are considered. The passed objectives can differ from the original number objectives.

Returns:

Merged costs.

Return type:

float

Raises:

RuntimeError – If the number of costs is different from the original number of objectives. If the objective was not found.

reset()

Reset the run to default values / empties.

Clear the initial data and configurations of the object.

Return type:

None

deepcave.runs.check_equality(runs, meta=False, configspace=True, objectives=True, budgets=True, seeds=False)

Check the passed runs on equality based on the selected runs.

Return the requested attributes.

Parameters:

• runs (list[AbstractRun]) – Runs to check for equality.

• meta (bool, optional) – Meta-Data excluding objectives and budgets, by default False.

• configspace (bool, optional) – Wheter to include the configuration space, by default True.

• objectives (bool, optional) – Wheter to include the objectives, by default True.

• budgets (bool, optional) – Whether to include the budgets, by default True.

• seeds (bool, optional) – Whether to include the seeds, by default False.

Returns:

Dictionary containing the checked attributes.

Return type:

Dict[str, Any]

Raises:

NotMergeableError – If the meta data of the runs are not equal. If the configuration spaces of the runs are not equal. If the budgets of the runs are not equal. If the objective of the runs are not equal.

Modules

deepcave.runs.converters	# converters
deepcave.runs.exceptions	# Exceptions
deepcave.runs.group	# Group
deepcave.runs.handler	# Handler
deepcave.runs.objective	# Objective
deepcave.runs.recorder	# Recorder
deepcave.runs.run	# Run
deepcave.runs.status	# Status
deepcave.runs.trial	# Trial

previous

API References

next

deepcave.runs.converters