cave.analyzer.cost_over_time module

class cave.analyzer.cost_over_time.CostOverTime(scenario, output_dir, rh: smac.runhistory.runhistory.RunHistory, runs: List[cave.reader.configurator_run.ConfiguratorRun], block_epm: bool = False, bohb_results=None, average_over_runs: bool = True, output_fn: str = 'performance_over_time.png', validator: Union[None, smac.utils.validate.Validator] = None)[source]

Bases: cave.analyzer.base_analyzer.BaseAnalyzer

Plot performance over time, using all trajectory entries where max_time = max(wallclock_limit, the highest recorded time)

Parameters
  • scenario (smac.scenario.scenario.Scenario) – scenario object with necessary information

  • output_dir (str) – output-directory for smac-object

  • rh (smac.runhistory.runhistory.RunHistory) – runhistory to use

  • runs (List[ConfiguratorRun]) – list of configurator-runs

  • block_epm (bool) – if block_epm, only use given runs to estimate cost

  • average_over_runs (bool) – if True, average over plots. if False, all runs are treated individually with checkboxes

  • output_fn (str) – path to output-png for this analysis

  • validator (Validator or None) – if given, use this epm to estimate costs for the individual incumbents (EPM)

get_html(d=None, tooltip=None)[source]

General reports in html-format, to be easily integrated in html-code. ALSO FOR BOKEH-OUTPUT.

Returns

script, div – header and body part of html-code

Return type

str, str

get_incumbent_trajectory(result, budgets, bigger_is_better=True, non_decreasing_budget=True)[source]

Returns the best configurations over time

!! Copied from hpbandster and modified to enable getting trajectories for individual budgets !!

Parameters
  • result – result

  • budgets (List[budgets]) – budgets to be considered

  • bigger_is_better (bool) – flag whether an evaluation on a larger budget is always considered better. If True, the incumbent might increase for the first evaluations on a bigger budget

  • non_decreasing_budget (bool) – flag whether the budget of a new incumbent should be at least as big as the one for the current incumbent.

Returns

dictionary with all the config IDs, the times the runs finished, their respective budgets, and corresponding losses

Return type

dict

get_jupyter()[source]

Depending on analysis, this creates jupyter-notebook compatible output.

get_plots()[source]
get_static_plots() → List[str]

Returns plot-paths, if any are available

Returns

plot_paths – returns list of strings

Return type

List[str]

get_table()

Get table, if available

plot()[source]

Plot performance over time, using all trajectory entries. max_time denotes max(wallclock_limit, highest recorded time).

class cave.analyzer.cost_over_time.Line(name, time, mean, upper, lower, config)

Bases: tuple

Create new instance of Line(name, time, mean, upper, lower, config)

config

Alias for field number 5

count(value) → integer -- return number of occurrences of value
index(value[, start[, stop]]) → integer -- return first index of value.

Raises ValueError if the value is not present.

lower

Alias for field number 4

mean

Alias for field number 2

name

Alias for field number 0

time

Alias for field number 1

upper

Alias for field number 3