cave.plot.configurator_footprint module

class cave.plot.configurator_footprint.ConfiguratorFootprintPlotter(scenario: smac.scenario.scenario.Scenario, rhs: smac.runhistory.runhistory.RunHistory, incs: list = None, final_incumbent=None, rh_labels=None, max_plot: int = -1, contour_step_size=0.2, use_timeslider: bool = False, num_quantiles: int = 10, timeslider_log: bool = True, output_dir: str = None)[source]

Bases: object

Creating an interactive plot, visualizing the configuration search space. The runhistories are correlated to the individual runs. Each run consists of a runhistory (in the smac-format), a list of incumbents If the dict “additional_info” in the RunValues of the runhistory contains a nested dict with additional_info[“timestamps”][“finished”], using those timestamps to sort data

Parameters:
  • scenario (Scenario) – scenario
  • rhs (List[RunHistory]) – runhistories from configurator runs, only data collected during optimization (no validation!)
  • incs (List[List[Configuration]]) – incumbents per run, last entry is final incumbent
  • final_incumbent (Configuration) – final configuration (best of all runs)
  • max_plot (int) – maximum number of configs to plot, if -1 plot all
  • contour_step_size (float) – step size of meshgrid to compute contour of fitness landscape
  • use_timeslider (bool) – whether or not to have a time_slider-widget on cfp-plot INCREASES FILE-SIZE DRAMATICALLY
  • num_quantiles (int) – number of quantiles for the slider/ number of static pictures
  • timeslider_log (bool) – whether to use a logarithmic scale for the timeslider/quantiles
  • output_dir (str) – output directory
get_conf_matrix(rh, incs)[source]

Iterates through runhistory to get a matrix of configurations (in vector representation), a list of configurations and the number of runs per configuration in a quantiled manner.

Parameters:
  • rh (RunHistory) – smac.runhistory
  • incs (List[List[Configuration]]) – incumbents of configurator runs, last entry is final incumbent
Returns:

  • conf_matrix (np.array) – matrix of configurations in vector representation
  • conf_list (np.array) – list of all Configuration objects that appeared in runhistory the order of this list is used to determine all kinds of properties in the plotting (but is arbitrarily determined)
  • runs_per_quantile (np.array) – numpy array of runs per configuration per quantile
  • labels (List[str]) – labels for timeslider (i.e. wallclock-times)

get_depth(cs: ConfigSpace.configuration_space.ConfigurationSpace, param: str)[source]

Get depth in configuration space of a given parameter name breadth search until reaching a leaf for the first time

Parameters:
  • cs (ConfigurationSpace) – ConfigurationSpace to get parents of a parameter
  • param (str) – name of parameter to inspect
get_distance(conf_matrix, cs: ConfigSpace.configuration_space.ConfigurationSpace)[source]

Computes the distance between all pairs of configurations.

Parameters:
  • conf_matrx (np.array) – numpy array with cols as parameter values
  • cs (ConfigurationSpace) – ConfigurationSpace to get conditionalities
Returns:

dists – np.array with distances between configurations i,j in dists[i,j] or dists[j,i]

Return type:

np.array

get_mds(dists)[source]

Compute multi-dimensional scaling (using sklearn MDS) – nonlinear scaling

Parameters:dists (np.array) – full matrix of distances between all configurations
Returns:scaled coordinates in 2-dim room
Return type:np.array
get_pred_surface(rh, X_scaled, conf_list: list, contour_step_size)[source]

fit epm on the scaled input dimension and return data to plot a contour plot of the empirical performance

Parameters:
  • rh (RunHistory) – runhistory
  • X_scaled (np.array) – configurations in scaled 2dim
  • conf_list (list) – list of Configuration objects
  • contour_step_size (float) – step-size for contour
Returns:

contour_data – x, y, Z for contour plots

Return type:

(np.array, np.array, np.array)

plot(X, conf_list: list, runs_per_quantile, inc_list: list = None, contour_data=None, use_timeslider=False, use_checkbox=True, timeslider_labels=None)[source]

plots sampled configuration in 2d-space; uses bokeh for interactive plot saves results in self.output, if set

Parameters:
  • X (np.array) – np.array with 2-d coordinates for each configuration
  • conf_list (list) – list of ALL configurations in the same order as X
  • runs_per_quantile (list[np.array]) – configurator-run to be analyzed, as a np.array with the number of target-algorithm-runs per config per quantile.
  • inc_list (list) – list of incumbents (Configuration)
  • contour_data (list) – contour data (xx,yy,Z)
  • use_timeslider (bool) – whether or not to have a time_slider-widget on cfp-plot INCREASES FILE-SIZE DRAMATICALLY
  • use_checkbox (bool) – have checkboxes to toggle individual runs
Returns:

  • (script, div) (str) – script and div of the bokeh-figure
  • over_time_paths (List[str]) – list with paths to the different quantiled timesteps of the configurator run (for static evaluation)

reduce_runhistory(rh: smac.runhistory.runhistory.RunHistory, max_configs: int, keep=None)[source]

Reduce configs to desired number, by default just drop the configs with the fewest runs.

Parameters:
  • rh (RunHistory) – runhistory that is to be reduced
  • max_configs (int) – if > -1 reduce runhistory to at most max_configs
  • keep (List[Configuration]) – list of configs that should be kept for sure (e.g. default, incumbents)
Returns:

rh – reduced runhistory

Return type:

RunHistory

run()[source]

Uses available Configurator-data to perform a MDS, estimate performance data and plot the configurator footprint.