cave.analyzer.performance_table module

class cave.analyzer.performance_table.PerformanceTable(instances: List[str], validated_rh: smac.runhistory.runhistory.RunHistory, default: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, epm_rh: smac.runhistory.runhistory.RunHistory, scenario: smac.scenario.scenario.Scenario, rng: mtrand.RandomState)[source]

Bases: cave.analyzer.base_analyzer.BaseAnalyzer

create_performance_table(default, incumbent, epm_rh, oracle)[source]

Create table, compare default against incumbent on train-, test- and combined instances. Listing PAR10, PAR1 and timeouts. Distinguishes between train and test, if available.

get_html(d=None, tooltip=None)[source]

General reports in html-format, to be easily integrated in html-code. ALSO FOR BOKEH-OUTPUT.

Returns:script, div – header and body part of html-code
Return type:str, str

Depending on analysis, this creates jupyter-notebook compatible output.

get_oracle(instances, rh)[source]

Estimation of oracle performance. Collects best performance seen for each instance in any run.

  • instances (List[str]) – list of instances in question
  • rh (RunHistory or List[RunHistory]) – runhistory or list of runhistories (will be combined)
  • Results
  • -------
  • oracle (dict[str->float]) – best seen performance per instance {inst : performance}
get_parX(cost_dict, par=10)[source]

Calculate parX-values from given cost_dict. First determine PAR-timeouts for each run on each instances, Second average over train/test if available, else just average.

  • cost_dict (Dict[inst->cost]) – mapping instances to costs
  • par (int) – par-factor to use

PAR10 values for train- and test-instances, if available as tuple else the general average

Return type:

(train, test) OR average – tuple<float, float> OR float

get_static_plots() → List[str]

Returns plot-paths, if any are available

Returns:plot_paths – returns list of strings
Return type:List[str]

Get table, if available


Get number of timeouts in config

Parameters:timeouts (dict[i -> bool]) – mapping instances to whether timeout was on that instance
Returns:timeouts – tuple (timeouts, total runs)
Return type:tuple(int, int)