smac.intensification.intensification module

class smac.intensification.intensification.Intensifier(stats: smac.stats.stats.Stats, traj_logger: smac.utils.io.traj_logging.TrajLogger, rng: numpy.random.mtrand.RandomState, instances: List[str], instance_specifics: Optional[Mapping[str, numpy.ndarray]] = None, cutoff: Optional[int] = None, deterministic: bool = False, run_obj_time: bool = True, always_race_against: Optional[ConfigSpace.configuration_space.Configuration] = None, run_limit: int = 2147483647, use_ta_time_bound: bool = False, minR: int = 1, maxR: int = 2000, adaptive_capping_slackfactor: float = 1.2, min_chall: int = 2)[source]

Bases: smac.intensification.abstract_racer.AbstractRacer

Races challengers against an incumbent

SMAC’s intensification procedure, in detail:

Procedure 2: Intensify(Θ_new, θ_inc, M, R, t_intensify, Π, cˆ) cˆ(θ, Π’) denotes the empirical cost of θ on the subset of instances Π’ ⊆ Π, based on the runs in R; maxR is a parameter where: Θ_new: Sequence of parameter settings to evaluate, challengers in this class. θ_inc: incumbent parameter setting, incumbent in this class.

1 for i := 1, … , length(Θ_new) do 2 θ_new ← Θ_new[i]

STAGE–>RUN_INCUMBENT

3 if R contains less than maxR runs with configuration θ_inc then 4 Π’ ← {π’∈ Π | R contains less than or equal number of runs using θ_inc and π’ 0 than using θ_inc and any other π’’∈ Π} 5 π ← instance sampled uniformly at random from Π’ 6 s ← seed, drawn uniformly at random 7 R ← ExecuteRun(R, θ_inc, π, s) 8 N ← 1

STAGE–>RUN_CHALLENGER

9 while true do 10 S_missing ← {instance, seed} pairs for which θ_inc was run before, but not θ_new 11 S_torun ← random subset of S_missing of size min(N, size(S_missing)) 12 foreach (π, s) ∈ S_torun do R ← ExecuteRun(R, θ_new, π, s) 13 S_missing ← S_missing S_torun 14 Π_common ← instances for which we previously ran both θ_inc and θ_new 15 if cˆ(θ_new, Π_common) > cˆ(θ_inc, Π_common) then break 16 else if S_missing = ∅ then θ_inc ← θ_new; break 17 else N ← 2 · N 18 if time spent in this call to this procedure exceeds t_intensify and i ≥ 2 then break 19 return [R, θ_inc]

Creates an Intensifier object

Parameters
  • stats (Stats) – stats object

  • traj_logger (TrajLogger) – TrajLogger object to log all new incumbents

  • rng (np.random.RandomState) –

  • instances (typing.List[str]) – list of all instance ids

  • instance_specifics (typing.Mapping[str,np.ndarray]) – mapping from instance name to instance specific string

  • cutoff (int) – runtime cutoff of TA runs

  • deterministic (bool) – whether the TA is deterministic or not

  • run_obj_time (bool) – whether the run objective is runtime or not (if true, apply adaptive capping)

  • always_race_against (Configuration) – if incumbent changes race this configuration always against new incumbent; can sometimes prevent over-tuning

  • use_ta_time_bound (bool,) – if true, trust time reported by the target algorithms instead of measuring the wallclock time for limiting the time of intensification

  • run_limit (int) – Maximum number of target algorithm runs per call to intensify.

  • maxR (int) – Maximum number of runs per config (summed over all calls to intensifiy).

  • minR (int) – Minimum number of run per config (summed over all calls to intensify).

  • adaptive_capping_slackfactor (float) – slack factor of adpative capping (factor * adpative cutoff)

  • min_chall (int) – minimal number of challengers to be considered (even if time_bound is exhausted earlier)

_generate_challengers(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser]) Iterator[Optional[ConfigSpace.configuration_space.Configuration]][source]

Retuns a sequence of challengers to use in intensification If challengers are not provided, then optimizer will be used to generate the challenger list

Parameters
Returns

A generator containing the next challengers to use

Return type

typing.Optional[typing.Generator[Configuration]]

_get_inc_available_inst(incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) List[str][source]

Implementation of line 4 of Intensification

This method queries the inc runs in the run history and return the pending instances if any is available

Parameters
  • incumbent (Configuration) – Either challenger or incumbent

  • run_history (RunHistory) – stores all runs we ran so far

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

_get_instances_to_run(challenger: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, N: int, run_history: smac.runhistory.runhistory.RunHistory) Tuple[List[smac.runhistory.runhistory.InstSeedBudgetKey], float][source]

Returns the minimum list of <instance, seed> pairs to run the challenger on before comparing it with the incumbent

Parameters
  • incumbent (Configuration) – incumbent configuration

  • challenger (Configuration) – promising configuration that is presently being evaluated

  • run_history (RunHistory) – Stores all runs we ran so far

  • N (int) – number of <instance, seed> pairs to select

Returns

  • typing.List[InstSeedBudgetKey] – list of <instance, seed, budget> tuples to run

  • float – total (runtime) cost of running the incumbent on the instances (used for adaptive capping while racing)

_get_next_inc_run(available_insts: List[str]) Tuple[str, int, Optional[float]][source]

Method to extract the next seed/instance in which a incumbent run most be evaluated.

Parameters

available_insts (typing.List[str]) – A list of instances from which to extract the next incumbent run

Returns

  • instance (str) – Next instance to evaluate

  • seed (float) – Seed in which to evaluate the instance

  • cutoff (Optional[float]) – Max time for a given instance/seed pair

_get_next_racer(challenger: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) Tuple[ConfigSpace.configuration_space.Configuration, str, int, Optional[float]][source]

Method to return the next config setting to aggressively race challenger against incumbent.

Parameters
  • challenger (Configuration) – Configuration which challenges incumbent

  • incumbent (Configuration) – Best configuration so far

  • run_history (RunHistory) – Stores all runs we ran so far

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

Returns

  • new_incumbent (Configuration) – Either challenger or incumbent

  • instance (str) – Next instance to evaluate

  • seed (int) – Seed in which to evaluate the instance

  • cutoff (Optional[float]) – Max time for a given instance/seed pair

_is_there_time_due_to_adaptive_cap(challenger: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory) bool[source]

A check to see if there is no more time for a challenger given the fact, that we are optimizing time and the incumbent looks more promising Line 18

Parameters
  • challenger (Configuration) – Configuration which challenges incumbent

  • run_history (RunHistory) – Stores all runs we ran so far

Returns

whether or not there is more time for a challenger run

Return type

bool

_next_iteration() None[source]

Updates tracking variables at the end of an intensification run

_process_inc_run(incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) None[source]

Method to process the results of a challenger that races an incumbent

Parameters
  • incumbent (Configuration) – Either challenger or incumbent

  • run_history (RunHistory) – stores all runs we ran so far

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

_process_racer_results(challenger: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) Optional[ConfigSpace.configuration_space.Configuration][source]

Process the result of a racing configuration against the current incumbent. Might propose a new incumbent.

Parameters
  • challenger (Configuration) – Configuration which challenges incumbent

  • incumbent (Configuration) – Best configuration so far

  • run_history (RunHistory) – Stores all runs we ran so far

Returns

new_incumbent – Either challenger or incumbent

Return type

typing.Optional[Configuration]

get_next_challenger(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser]) Tuple[Optional[ConfigSpace.configuration_space.Configuration], bool][source]

This function returns the next challenger, that should be exercised though lines 8-17.

It does so by populating configs_to_run, which is a pool of configuration from which the racer will sample. Each configuration within configs_to_run, will be intensified on different instances/seed registered in self.to_run as stated in line 11.

A brand new configuration should only be sampled, after all self.to_run instance seed pairs are exhausted.

This method triggers a call to _next_iteration if there are no more configurations to run, for the current intensification loop. This marks the transition to Line 2, where a new configuration to intensify will be drawn from epm/initial challengers.

Parameters
Returns

  • typing.Optional[Configuration] – next configuration to evaluate

  • bool – flag telling if the configuration is newly sampled or one currently being tracked

get_next_run(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], incumbent: ConfigSpace.configuration_space.Configuration, chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser], run_history: smac.runhistory.runhistory.RunHistory, repeat_configs: bool = True, num_workers: int = 1) Tuple[smac.intensification.abstract_racer.RunInfoIntent, smac.runhistory.runhistory.RunInfo][source]

This procedure is in charge of generating a RunInfo object to comply with lines 7 (in case stage is stage==RUN_INCUMBENT) or line 12 (In case of stage==RUN_CHALLENGER)

A RunInfo object encapsulates the necessary information for a worker to execute the job, nevertheless, a challenger is not always available. This could happen because no more configurations are available or the new configuration to try was already executed.

To circumvent this, a intent is also returned:

  • (intent=RUN) Run the RunInfo object (Normal Execution

  • (intent=SKIP) Skip this iteration. No challenger is available, in particular

    because challenger is the same as incumbent

Parameters
  • challengers (typing.List[Configuration]) – promising configurations

  • incumbent (Configuration) – incumbent configuration

  • chooser (smac.optimizer.epm_configuration_chooser.EPMChooser) – optimizer that generates next configurations to use for racing

  • run_history (RunHistory) – stores all runs we ran so far

  • repeat_configs (bool) – if False, an evaluated configuration will not be generated again

  • num_workers (int) – the maximum number of workers available at a given time.

Returns

  • intent (RunInfoIntent) – What should the smbo object do with the runinfo.

  • run_info (RunInfo) – An object that encapsulates necessary information for a config run

process_results(run_info: smac.runhistory.runhistory.RunInfo, incumbent: Optional[ConfigSpace.configuration_space.Configuration], run_history: smac.runhistory.runhistory.RunHistory, time_bound: float, result: smac.runhistory.runhistory.RunValue, log_traj: bool = True) Tuple[ConfigSpace.configuration_space.Configuration, float][source]

The intensifier stage will be updated based on the results/status of a configuration execution.

During intensification, the following can happen:

  • Challenger raced against incumbent

  • Also, during a challenger run, a capped exception can be triggered, where no racer post processing is needed

  • A run on the incumbent for more confidence needs to be processed, IntensifierStage.PROCESS_INCUMBENT_RUN

  • The first run results need to be processed (PROCESS_FIRST_CONFIG_RUN)

At the end of any run, checks are done to move to a new iteration.

Parameters
  • run_info (RunInfo) – A RunInfo containing the configuration that was evaluated

  • incumbent (typing.Optional[Configuration]) – best configuration so far, None in 1st run

  • run_history (RunHistory) – stores all runs we ran so far if False, an evaluated configuration will not be generated again

  • time_bound (float) – time in [sec] available to perform intensify

  • result (RunValue) – Contain the result (status and other methadata) of exercising a challenger/incumbent.

  • log_traj (bool) – whether to log changes of incumbents in trajectory

Returns

  • incumbent (Configuration()) – current (maybe new) incumbent configuration

  • inc_perf (float) – empirical performance of incumbent configuration

class smac.intensification.intensification.IntensifierStage(value)[source]

Bases: enum.Enum

Class to define different stages of intensifier

PROCESS_FIRST_CONFIG_RUN = 4
PROCESS_INCUMBENT_RUN = 5
RUN_BASIS = 3
RUN_CHALLENGER = 2
RUN_FIRST_CONFIG = 0
RUN_INCUMBENT = 1
exception smac.intensification.intensification.NoMoreChallengers[source]

Bases: Exception

Indicates that no more challengers are available for the intensification to proceed