cave.reader.configurator_run module

class cave.reader.configurator_run.ConfiguratorRun(scenario, original_runhistory, validated_runhistory, trajectory, options, path_to_folder=None, ta_exec_dir=None, file_format=None, validation_format=None, reduced_to_budgets=None, output_dir=None)

Bases: object

ConfiguratorRuns load and maintain information about individual configurator runs. There are different supported formats, like: BOHB, SMAC3, SMAC2 and CSV This class is responsible for providing a scenario, a runhistory and a trajectory and handling original/validated data appropriately. To create a ConfiguratorRun from a folder, use Configurator.from_folder()

Parameters

• scenario (Scenario) – scenario

• validated_runhistory (original_runhistory,) – runhistores containing only the original evaluated data (during optimization process) or the validated data where points of interest are reevaluated after the optimization process

• trajectory (List[dict]) – a trajectory of the best performing configurations at each point in time

• options (dict) – options can define a number of custom settings

• path_to_folder (str) – path to the physical folder containing the data

• ta_exec_dir (str) – path to the target-algorithm-execution-directory. This is only important for SMAC-optimized data

• validation_format (file_format,) – will be autodetected some point soon, until then, specify the file-format (SMAC2, SMAC3, BOHB, etc…)

• reduced_to_budgets (List str int or float) – budgets, with which this cr is associated

• output_dir (str) – where to save analysis-data for this cr

_check_rh_for_inc_and_def(rh, name='')

Check if default and incumbent are evaluated on all instances in this rh

Parameters

• rh (RunHistory) – runhistory to be checked

• name (str) – name for logging-purposes

Returns

return_value – False if either inc or def was not evaluated on all train/test-instances

Return type

bool

_get_feature_names()

_init_pimp_and_validator(alternative_output_dir=None)

Create ParameterImportance-object and use it’s trained model for validation and further predictions. We pass a combined (original + validated) runhistory, so that the returned model will be based on as much information as possible

Parameters

alternative_output_dir (str) – e.g. for budgets we want pimp to use an alternative output-dir (subfolders per budget)

_validate_default_and_incumbents(method, ta_exec_dir)

Validate default and incumbent configurations on all instances possible. Either use validation (physically execute the target algorithm) or EPM-estimate and update according runhistory (validation -> self.global_validated_rh; epm -> self.global_epm_rh).

Parameters

• method (str) – epm or validation

• ta_exec_dir (str) – path from where the target algorithm can be executed as found in scenario (only used for actual validation)

classmethod from_folder(folder: str, ta_exec_dir: str, options, file_format: str = 'SMAC3', validation_format: str = 'NONE', output_dir=None)

Initialize scenario, runhistory and incumbent from folder

Parameters

• folder (string) – output-dir of this configurator-run -> this is also the ‘id’ for a single run in parallel optimization

• ta_exec_dir (string) – if the execution directory for the SMAC-run differs from the cwd, there might be problems loading instance-, feature- or PCS-files in the scenario-object. since instance- and PCS-files are necessary, specify the path to the execution-dir of SMAC here

• file_format (string) – from [SMAC2, SMAC3, BOHB, APT, CSV]

• validation_format (string) – from [SMAC2, SMAC3, APT, CSV, NONE], in which format to look for validated data

get_budgets()

get_identifier()

get_incumbent()

classmethod get_reader(name, folder, ta_exec_dir)

Returns an appropriate reader for the specified format.

classmethod identify(path, budget)

cave.reader.configurator_run._changedir(newdir)

Helper function to change directory, for example to create a scenario from file, where paths to the instance- and feature-files are relative to the original SMAC-execution-directory. Same with target algorithms that need be executed for validation.