cave.reader.configurator_run module

class cave.reader.configurator_run.ConfiguratorRun(scenario, original_runhistory, validated_runhistory, trajectory, options, path_to_folder=None, ta_exec_dir=None, file_format=None, validation_format=None, reduced_to_budgets=None, output_dir=None)[source]

Bases: object

ConfiguratorRuns load and maintain information about individual configurator runs. There are different supported formats, like: BOHB, SMAC3, SMAC2 and CSV This class is responsible for providing a scenario, a runhistory and a trajectory and handling original/validated data appropriately. To create a ConfiguratorRun from a folder, use Configurator.from_folder()

  • scenario (Scenario) – scenario

  • validated_runhistory (original_runhistory,) – runhistores containing only the original evaluated data (during optimization process) or the validated data where points of interest are reevaluated after the optimization process

  • trajectory (List[dict]) – a trajectory of the best performing configurations at each point in time

  • options (dict) – options can define a number of custom settings

  • path_to_folder (str) – path to the physical folder containing the data

  • ta_exec_dir (str) – path to the target-algorithm-execution-directory. This is only important for SMAC-optimized data

  • validation_format (file_format,) – will be autodetected some point soon, until then, specify the file-format (SMAC2, SMAC3, BOHB, etc…)

  • reduced_to_budgets (List str int or float) – budgets, with which this cr is associated

  • output_dir (str) – where to save analysis-data for this cr

_check_rh_for_inc_and_def(rh, name='')[source]

Check if default and incumbent are evaluated on all instances in this rh

  • rh (RunHistory) – runhistory to be checked

  • name (str) – name for logging-purposes


return_value – False if either inc or def was not evaluated on all train/test-instances

Return type



Create ParameterImportance-object and use it’s trained model for validation and further predictions. We pass a combined (original + validated) runhistory, so that the returned model will be based on as much information as possible


alternative_output_dir (str) – e.g. for budgets we want pimp to use an alternative output-dir (subfolders per budget)

_validate_default_and_incumbents(method, ta_exec_dir)[source]

Validate default and incumbent configurations on all instances possible. Either use validation (physically execute the target algorithm) or EPM-estimate and update according runhistory (validation -> self.global_validated_rh; epm -> self.global_epm_rh).

  • method (str) – epm or validation

  • ta_exec_dir (str) – path from where the target algorithm can be executed as found in scenario (only used for actual validation)

classmethod from_folder(folder: str, ta_exec_dir: str, options, file_format: str = 'SMAC3', validation_format: str = 'NONE', output_dir=None)[source]

Initialize scenario, runhistory and incumbent from folder

  • folder (string) – output-dir of this configurator-run -> this is also the ‘id’ for a single run in parallel optimization

  • ta_exec_dir (string) – if the execution directory for the SMAC-run differs from the cwd, there might be problems loading instance-, feature- or PCS-files in the scenario-object. since instance- and PCS-files are necessary, specify the path to the execution-dir of SMAC here

  • file_format (string) – from [SMAC2, SMAC3, BOHB, APT, CSV]

  • validation_format (string) – from [SMAC2, SMAC3, APT, CSV, NONE], in which format to look for validated data

classmethod get_reader(name, folder, ta_exec_dir)[source]

Returns an appropriate reader for the specified format.

classmethod identify(path, budget)[source]

Helper function to change directory, for example to create a scenario from file, where paths to the instance- and feature-files are relative to the original SMAC-execution-directory. Same with target algorithms that need be executed for validation.