smac.intensification.hyperband module

class smac.intensification.hyperband.Hyperband(stats: smac.stats.stats.Stats, traj_logger: smac.utils.io.traj_logging.TrajLogger, rng: numpy.random.mtrand.RandomState, instances: List[str], instance_specifics: Mapping[str, numpy.ndarray] = None, cutoff: Optional[float] = None, deterministic: bool = False, initial_budget: Optional[float] = None, max_budget: Optional[float] = None, eta: float = 3, run_obj_time: bool = True, n_seeds: Optional[int] = None, instance_order: str = 'shuffle_once', adaptive_capping_slackfactor: float = 1.2, min_chall: int = 1, incumbent_selection: str = 'highest_executed_budget')[source]

Bases: smac.intensification.parallel_scheduling.ParallelScheduler

Races multiple challengers against an incumbent using Hyperband method

Implementation from “BOHB: Robust and Efficient Hyperparameter Optimization at Scale” (Falkner et al. 2018)

Hyperband is an extension of the Successive Halving intensifier. Please refer to SuccessiveHalving documentation for more detailed information about the different types of budgets possible and the way instances are handled.

Internally, this class uses the _Hyperband private class which actually implements the hyperband logic. To allow for parallelism, Hyperband can create multiple _Hyperband instances, based on the number of idle workers available.

Parameters
  • stats (smac.stats.stats.Stats) – stats object

  • traj_logger (smac.utils.io.traj_logging.TrajLogger) – TrajLogger object to log all new incumbents

  • rng (np.random.RandomState) –

  • instances (typing.List[str]) – list of all instance ids

  • instance_specifics (typing.Mapping[str,np.ndarray]) – mapping from instance name to instance specific string

  • cutoff (typing.Optional[int]) – runtime cutoff of TA runs

  • deterministic (bool) – whether the TA is deterministic or not

  • initial_budget (typing.Optional[float]) – minimum budget allowed for 1 run of successive halving

  • max_budget (typing.Optional[float]) – maximum budget allowed for 1 run of successive halving

  • eta (float) – ‘halving’ factor after each iteration in a successive halving run. Defaults to 3

  • run_obj_time (bool) – whether the run objective is runtime or not (if true, apply adaptive capping)

  • n_seeds (typing.Optional[int]) – Number of seeds to use, if TA is not deterministic. Defaults to None, i.e., seed is set as 0

  • instance_order (typing.Optional[str]) – how to order instances. Can be set to: [None, shuffle_once, shuffle] * None - use as is given by the user * shuffle_once - shuffle once and use across all SH run (default) * shuffle - shuffle before every SH run

  • adaptive_capping_slackfactor (float) – slack factor of adpative capping (factor * adpative cutoff)

  • min_chall (int) – minimal number of challengers to be considered (even if time_bound is exhausted earlier). This class will raise an exception if a value larger than 1 is passed.

  • incumbent_selection (str) – How to select incumbent in successive halving. Only active for real-valued budgets. Can be set to: [highest_executed_budget, highest_budget, any_budget] * highest_executed_budget - incumbent is the best in the highest budget run so far (default) * highest_budget - incumbent is selected only based on the highest budget * any_budget - incumbent is the best on any budget i.e., best performance regardless of budget

_add_new_instance(num_workers: int) → bool[source]

Decides if it is possible to add a new intensifier instance, and adds it. If a new intensifier instance is added, True is returned, else False.

num_workers: int

the maximum number of workers available at a given time.

Returns

Return type

Whether or not a new instance was added

_get_intensifier_ranking(intensifier: smac.intensification.abstract_racer.AbstractRacer) → Tuple[int, int][source]

Given a intensifier, returns how advance it is. This metric will be used to determine what priority to assign to the intensifier

Parameters

intensifier (AbstractRacer) – Intensifier to rank based on run progress

Returns

  • ranking (int) – the higher this number, the faster the intensifier will get the running resources. For hyperband we can use the sh_intensifier stage, for example

  • tie_breaker (int) – The configurations that have been launched to break ties. For example, in the case of Successive Halving it can be the number of configurations launched

class smac.intensification.hyperband._Hyperband(stats: smac.stats.stats.Stats, traj_logger: smac.utils.io.traj_logging.TrajLogger, rng: numpy.random.mtrand.RandomState, instances: List[str], instance_specifics: Mapping[str, numpy.ndarray] = None, cutoff: Optional[float] = None, deterministic: bool = False, initial_budget: Optional[float] = None, max_budget: Optional[float] = None, eta: float = 3, run_obj_time: bool = True, n_seeds: Optional[int] = None, instance_order: str = 'shuffle_once', adaptive_capping_slackfactor: float = 1.2, min_chall: int = 1, incumbent_selection: str = 'highest_executed_budget', identifier: int = 0)[source]

Bases: smac.intensification.successive_halving._SuccessiveHalving

Races multiple challengers against an incumbent using Hyperband method

This class contains the logic to implement: “BOHB: Robust and Efficient Hyperparameter Optimization at Scale” (Falkner et al. 2018)

Objects from this class are meant to run on a single worker. Hyperband method, creates multiple _Hyperband instances to allow parallelism, and for this reason Hyperband should be considered the user interface whereas _Hyperband a private class with the actual implementation of the method.

Parameters
  • stats (smac.stats.stats.Stats) – stats object

  • traj_logger (smac.utils.io.traj_logging.TrajLogger) – TrajLogger object to log all new incumbents

  • rng (np.random.RandomState) –

  • instances (typing.List[str]) – list of all instance ids

  • instance_specifics (typing.Mapping[str,np.ndarray]) – mapping from instance name to instance specific string

  • cutoff (typing.Optional[int]) – runtime cutoff of TA runs

  • deterministic (bool) – whether the TA is deterministic or not

  • initial_budget (typing.Optional[float]) – minimum budget allowed for 1 run of successive halving

  • max_budget (typing.Optional[float]) – maximum budget allowed for 1 run of successive halving

  • eta (float) – ‘halving’ factor after each iteration in a successive halving run. Defaults to 3

  • run_obj_time (bool) – whether the run objective is runtime or not (if true, apply adaptive capping)

  • n_seeds (typing.Optional[int]) – Number of seeds to use, if TA is not deterministic. Defaults to None, i.e., seed is set as 0

  • instance_order (typing.Optional[str]) – how to order instances. Can be set to: [None, shuffle_once, shuffle] * None - use as is given by the user * shuffle_once - shuffle once and use across all SH run (default) * shuffle - shuffle before every SH run

  • adaptive_capping_slackfactor (float) – slack factor of adpative capping (factor * adpative cutoff)

  • min_chall (int) – minimal number of challengers to be considered (even if time_bound is exhausted earlier). This class will raise an exception if a value larger than 1 is passed.

  • incumbent_selection (str) – How to select incumbent in successive halving. Only active for real-valued budgets. Can be set to: [highest_executed_budget, highest_budget, any_budget] * highest_executed_budget - incumbent is the best in the highest budget run so far (default) * highest_budget - incumbent is selected only based on the highest budget * any_budget - incumbent is the best on any budget i.e., best performance regardless of budget

  • identifier (int) – Allows to identify the _Hyperband instance in case of multiple ones

_update_stage(run_history: smac.runhistory.runhistory.RunHistory = None) → None[source]

Update tracking information for a new stage/iteration and update statistics. This method is called to initialize stage variables and after all configurations of a successive halving stage are completed.

Parameters

run_history (smac.runhistory.runhistory.RunHistory) – stores all runs we ran so far

get_next_run(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], incumbent: ConfigSpace.configuration_space.Configuration, chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser], run_history: smac.runhistory.runhistory.RunHistory, repeat_configs: bool = True, num_workers: int = 1) → Tuple[smac.intensification.abstract_racer.RunInfoIntent, smac.runhistory.runhistory.RunInfo][source]

Selects which challenger to use based on the iteration stage and set the iteration parameters. First iteration will choose configurations from the chooser or input challengers, while the later iterations pick top configurations from the previously selected challengers in that iteration

If no new run is available, the method returns a configuration of None.

Parameters
Returns

  • intent (RunInfoIntent) – Indicator of how to consume the RunInfo object

  • run_info (RunInfo) – An object that encapsulates necessary information for a config run

process_results(run_info: smac.runhistory.runhistory.RunInfo, incumbent: Optional[ConfigSpace.configuration_space.Configuration], run_history: smac.runhistory.runhistory.RunHistory, time_bound: float, result: smac.runhistory.runhistory.RunValue, log_traj: bool = True) → Tuple[ConfigSpace.configuration_space.Configuration, float][source]

The intensifier stage will be updated based on the results/status of a configuration execution. Also, a incumbent will be determined.

Parameters
  • run_info (RunInfo) – A RunInfo containing the configuration that was evaluated

  • incumbent (typing.Optional[Configuration]) – Best configuration seen so far

  • run_history (RunHistory) – stores all runs we ran so far if False, an evaluated configuration will not be generated again

  • time_bound (float) – time in [sec] available to perform intensify

  • result (RunValue) – Contain the result (status and other methadata) of exercising a challenger/incumbent.

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

Returns

  • incumbent (Configuration) – current (maybe new) incumbent configuration

  • inc_perf (float) – empirical performance of incumbent configuration