cave.analyzer.performance_table module

class cave.analyzer.performance_table.PerformanceTable(instances: List[str], validated_rh: smac.runhistory.runhistory.RunHistory, default: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, epm_rh: smac.runhistory.runhistory.RunHistory, scenario: smac.scenario.scenario.Scenario, rng: mtrand.RandomState)[source]

Bases: cave.analyzer.base_analyzer.BaseAnalyzer

create_performance_table(default, incumbent, epm_rh, oracle)[source]

Create table, compare default against incumbent on train-, test- and combined instances. Listing PAR10, PAR1 and timeouts. Distinguishes between train and test, if available.

get_html(d=None, tooltip=None)[source]

General reports in html-format, to be easily integrated in html-code. ALSO FOR BOKEH-OUTPUT.


script, div – header and body part of html-code

Return type

str, str


Depending on analysis, this creates jupyter-notebook compatible output.

get_oracle(instances, rh)[source]

Estimation of oracle performance. Collects best performance seen for each instance in any run.

  • instances (List[str]) – list of instances in question

  • rh (RunHistory or List[RunHistory]) – runhistory or list of runhistories (will be combined)

  • Results

  • -------

  • oracle (dict[str->float]) – best seen performance per instance {inst : performance}

get_parX(cost_dict, par=10)[source]

Calculate parX-values from given cost_dict. First determine PAR-timeouts for each run on each instances, Second average over train/test if available, else just average.

  • cost_dict (Dict[inst->cost]) – mapping instances to costs

  • par (int) – par-factor to use


PAR10 values for train- and test-instances, if available as tuple else the general average

Return type

(train, test) OR average – tuple<float, float> OR float

get_static_plots() → List[str]

Returns plot-paths, if any are available


plot_paths – returns list of strings

Return type



Get table, if available


Get number of timeouts in config


timeouts (dict[i -> bool]) – mapping instances to whether timeout was on that instance


timeouts – tuple (timeouts, total runs)

Return type

tuple(int, int)