cave.cavefacade module

class cave.cavefacade.CAVE(folders: List[str], output_dir: str, ta_exec_dir: List[str], file_format: str = 'auto', validation_format='NONE', validation_method: str = 'epm', seed: int = 42, show_jupyter: bool = True, verbose_level: str = 'OFF', analyzing_options=None, **kwargs)[source]

Bases: object

Initialize CAVE facade to handle analyzing, plotting and building the report-page easily. During initialization, the analysis-infrastructure is built and the data is validated, the overall best incumbent is found and default+incumbent are evaluated for all instances for all runs, by default using an EPM.

Parameters

  • folders (list<strings>) – paths to relevant SMAC runs

  • output_dir (string) – output for cave to write results (figures + report)

  • ta_exec_dir (string) – execution directory for target algorithm (to find instance.txt specified in scenario, ..)

  • file_format (string) – what format the rundata is in, options are [SMAC3, SMAC2, BOHB and CSV]

  • validation_format (string) – what format the validation rundata is in, options are [SMAC3, SMAC2, CSV and None]

  • validation_method (string) – from [validation, epm], how to estimate missing runs

  • seed (int) – random seed for analysis (e.g. the random forests)

  • show_jupyter (bool) – default True, tries to output plots and tables to jupyter-frontend, if available

  • verbose_level (string) – from [OFF, INFO, DEBUG, DEV_DEBUG and WARNING]

  • analyzing_options (string or dict) – options-dictionary following CAVE’s options-syntax

_build_website()[source]
_create_outputdir(output_dir)[source]

Creates output-dir, if necessary. Also sets the ‘self.output_dir_created’-flag, so this only happens once. If there is a directory already, zip this into an archive in the output_dir called ‘.OLD.zip’.

_get_dict(d, layername)[source]

Get the appropriate sub-dict for this layer (or layer-run combination) and create it if necessary

algorithm_footprints()[source]
analyze(options=None)[source]

Analyze the available data and build HTML-webpage as dict. Save webpage in ‘self.output_dir/CAVE/report.html’.

Parameters

options (Dict or str) – either a dictionary or a path to an ini-file.

apt_overview()[source]
apt_tensorboard()[source]
bohb_incumbents_per_budget()[source]
bohb_learning_curves()[source]
box_violin()[source]
budget_correlation()[source]
cave_ablation()[source]
cave_fanova()[source]
compare_default_incumbent()[source]
configurator_footprint(time_slider=None, max_configurations_to_plot=None, number_quantiles=None)[source]
cost_over_time(incumbent_trajectory=None)[source]
feature_analysis(d)[source]
feature_clustering()[source]
feature_correlation()[source]
feature_importance()[source]
local_parameter_importance()[source]
overview_table()[source]
parallel_coordinates(pc_sort_by: str = None, params: Union[int, List[str]] = None, n_configs: int = None, max_runs_epm: int = None)[source]
parameter_importance(d)[source]

Perform the specified parameter importance procedures.

performance_table()[source]
pimp_comparison_table(pimp_sort_table_by='average')[source]
pimp_forward_selection()[source]
plot_ecdf()[source]
plot_scatter()[source]
print_budgets()[source]

If the analyzed configurator uses budgets, print a list of available budgets.

set_verbosity(level)[source]

Set verbosity level and configure loggers (to stdout and to file), create output-dir if necessary.

Parameters

level (str) – From [INFO, WARNING, OFF, DEBUG, DEV_DEBUG]

cave.cavefacade._analyzer_type(f)[source]