cave.analyzer.performance_table module

class cave.analyzer.performance_table.PerformanceTable(instances: List[str], validated_rh: smac.runhistory.runhistory.RunHistory, default: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, epm_rh: smac.runhistory.runhistory.RunHistory, scenario: smac.scenario.scenario.Scenario, rng: mtrand.RandomState)[source]

Bases: cave.analyzer.base_analyzer.BaseAnalyzer

create_performance_table(default, incumbent, epm_rh, oracle)[source]

Create table, compare default against incumbent on train-, test- and combined instances. Listing PAR10, PAR1 and timeouts. Distinguishes between train and test, if available.

get_html(d=None, tooltip=None)[source]

General reports in html-format, to be easily integrated in html-code. ALSO FOR BOKEH-OUTPUT.

Returns

script, div – header and body part of html-code

Return type

str, str

get_jupyter()[source]

Depending on analysis, this creates jupyter-notebook compatible output.

get_oracle(instances, rh)[source]

Estimation of oracle performance. Collects best performance seen for each instance in any run.

Parameters
  • instances (List[str]) – list of instances in question

  • rh (RunHistory or List[RunHistory]) – runhistory or list of runhistories (will be combined)

  • Results

  • -------

  • oracle (dict[str->float]) – best seen performance per instance {inst : performance}

get_parX(cost_dict, par=10)[source]

Calculate parX-values from given cost_dict. First determine PAR-timeouts for each run on each instances, Second average over train/test if available, else just average.

Parameters
  • cost_dict (Dict[inst->cost]) – mapping instances to costs

  • par (int) – par-factor to use

Returns

PAR10 values for train- and test-instances, if available as tuple else the general average

Return type

(train, test) OR average – tuple<float, float> OR float

get_static_plots() → List[str]

Returns plot-paths, if any are available

Returns

plot_paths – returns list of strings

Return type

List[str]

get_table()

Get table, if available

timeouts_to_tuple(timeouts)[source]

Get number of timeouts in config

Parameters

timeouts (dict[i -> bool]) – mapping instances to whether timeout was on that instance

Returns

timeouts – tuple (timeouts, total runs)

Return type

tuple(int, int)