cave.analyzer.cost_over_time module

class cave.analyzer.cost_over_time.CostOverTime(scenario, output_dir, rh: smac.runhistory.runhistory.RunHistory, runs: List[cave.reader.configurator_run.ConfiguratorRun], block_epm: bool = False, bohb_results=None, average_over_runs: bool = True, output_fn: str = 'performance_over_time.png', validator: Union[None, smac.utils.validate.Validator] = None)[source]

Bases: cave.analyzer.base_analyzer.BaseAnalyzer

Plot performance over time, using all trajectory entries where max_time = max(wallclock_limit, the highest recorded time)

  • scenario (smac.scenario.scenario.Scenario) – scenario object with necessary information

  • output_dir (str) – output-directory for smac-object

  • rh (smac.runhistory.runhistory.RunHistory) – runhistory to use

  • runs (List[ConfiguratorRun]) – list of configurator-runs

  • block_epm (bool) – if block_epm, only use given runs to estimate cost

  • average_over_runs (bool) – if True, average over plots. if False, all runs are treated individually with checkboxes

  • output_fn (str) – path to output-png for this analysis

  • validator (Validator or None) – if given, use this epm to estimate costs for the individual incumbents (EPM)

get_html(d=None, tooltip=None)[source]

General reports in html-format, to be easily integrated in html-code. ALSO FOR BOKEH-OUTPUT.


script, div – header and body part of html-code

Return type

str, str

get_incumbent_trajectory(result, budgets, bigger_is_better=True, non_decreasing_budget=True)[source]

Returns the best configurations over time

!! Copied from hpbandster and modified to enable getting trajectories for individual budgets !!

  • result – result

  • budgets (List[budgets]) – budgets to be considered

  • bigger_is_better (bool) – flag whether an evaluation on a larger budget is always considered better. If True, the incumbent might increase for the first evaluations on a bigger budget

  • non_decreasing_budget (bool) – flag whether the budget of a new incumbent should be at least as big as the one for the current incumbent.


dictionary with all the config IDs, the times the runs finished, their respective budgets, and corresponding losses

Return type



Depending on analysis, this creates jupyter-notebook compatible output.

get_static_plots() → List[str]

Returns plot-paths, if any are available


plot_paths – returns list of strings

Return type



Get table, if available


Plot performance over time, using all trajectory entries. max_time denotes max(wallclock_limit, highest recorded time).

class cave.analyzer.cost_over_time.Line(name, time, mean, upper, lower, config)

Bases: tuple

Create new instance of Line(name, time, mean, upper, lower, config)


Alias for field number 5

count(value) → integer -- return number of occurrences of value
index(value[, start[, stop]]) → integer -- return first index of value.

Raises ValueError if the value is not present.


Alias for field number 4


Alias for field number 2


Alias for field number 0


Alias for field number 1


Alias for field number 3