cave.reader.runs_container module

class cave.reader.runs_container.RunsContainer(folders, ta_exec_dirs=None, output_dir=None, file_format=None, validation_format=None, analyzing_options=None)

Bases: object

Reads in optimizer runs. Converts data if necessary.

SMAC3’s RunHistory supports budgets from 0.12.0, so this container will by default keep one ConfiguratorRun per folder (assuming folders are parallel runs). Budgets are integrated in RunHistories per conversion. The RunHistory object provides an easy way to aggregate over parallel runs or budgets.

The data is organized in self.data as {folder_name : ConfiguratorRun}. Aggregated or reduced ConfiguratorRuns are cached by their identifier (needs to be unique from context!)

in self.cache as {identifier : ConfiguratorRun},-

In the internal data-management there are three types of runhistories: original, validated and epm. They are saved in and provided by the ConfiguratorRuns

• original_rh contain only runs that have been gathered during the optimization-process.

• validated_rh may contain original runs, but also data that was not gathered iteratively during the optimization, but systematically through external validation of interesting configurations. Important: NO ESTIMATED RUNS IN validated RUNHISTORIES!

• epm_rh contain runs that are gathered through empirical performance models.

Runhistories are organized as follows:

• each ConfiguratorRun has an original_runhistory- and a combined_runhistory-attribute

• if available, each ConfiguratorRun’s validated_runhistory contains a runhistory with validation-data gathered after the optimization

• combined_runhistory always contains as many real runs as possible

Parameters

• folders (List[str]) – list of folders to read in

• ta_exec_dirs (List[str]) – optional, list of execution directories for target-algorithms (to find filepaths, etc.). If you’re not sure, just set to current working directory (which it is by default).

• output_dir (str) – directory for output (temporary directory if not set)

• file_format (str) – optional, from [AUTO, SMAC2, SMAC3, BOHB, CSV] defines what file-format the optimizer result is in. AUTO or None will lead to attempted automatic detection

• validation_format (str) – from [SMAC2, SMAC3, BOHB, CSV] defines what file-format validation data is in.

• analyzing_options (dict / ConfigParser) – contains important global configurations on how to run CAVE, see options

_aggregate(runs)

_cache(configurator_run)

_reduce_cr_to_budget(cr, keep_budgets)

Creates a new ConfiguratorRun without all the target algorithm runs that are not in the list of budgets. Will affect original, validated and epm-RunHistories as well as Trajectory

get_aggregated(keep_budgets=True, keep_folders=False)

Collapse data-structure along a given “axis”.

Returns

aggregated_runs – run(s) with aggregated data

Return type

List[ConfiguratorRun]

get_all_runs()

get_budgets()

get_folders()

get_highest_budget()

get_rng()

get_run(folder, budget)

get_runs_for_budget(target_b)

get_runs_for_folder(f)