smac.intensification.intensification module

class smac.intensification.intensification.Intensifier(stats: smac.stats.stats.Stats, traj_logger: smac.utils.io.traj_logging.TrajLogger, rng: numpy.random.mtrand.RandomState, instances: List[str], instance_specifics: Mapping[str, numpy.ndarray] = None, cutoff: int = None, deterministic: bool = False, run_obj_time: bool = True, always_race_against: ConfigSpace.configuration_space.Configuration = None, run_limit: int = 2147483647, use_ta_time_bound: bool = False, minR: int = 1, maxR: int = 2000, adaptive_capping_slackfactor: float = 1.2, min_chall: int = 2)[source]

Bases: smac.intensification.abstract_racer.AbstractRacer

Races challengers against an incumbent

SMAC’s intensification procedure, in detail:

Procedure 2: Intensify(Θ_new, θ_inc, M, R, t_intensify, Π, cˆ) cˆ(θ, Π’) denotes the empirical cost of θ on the subset of instances Π’ ⊆ Π, based on the runs in R; maxR is a parameter where: Θ_new: Sequence of parameter settings to evaluate, challengers in this class. θ_inc: incumbent parameter setting, incumbent in this class. ____________________________________________________________ 1 for i := 1, … , length(Θ_new) do 2 θ_new ← Θ_new[i]

3 if R contains less than maxR runs with configuration θ_inc then 4 Π’ ← {π’∈ Π | R contains less than or equal number of runs using θ_inc and π’ 0 than using θ_inc and any other π’’∈ Π} 5 π ← instance sampled uniformly at random from Π’ 6 s ← seed, drawn uniformly at random 7 R ← ExecuteRun(R, θ_inc, π, s) 8 N ← 1

9 while true do

10 S_missing ← {instance, seed} pairs for which θ_inc was run before, but not θ_new 11 S_torun ←random subset of S_missing of size min(N, |S_missing|) 12 foreach (π, s) ∈ S_torun do R ← ExecuteRun(R, θ_new, π, s) 13 S_missing ← S_missing S_torun 14 Π_common ← instances for which we previously ran both θ_inc and θ_new 15 if cˆ(θ_new, Π_common) > cˆ(θ_inc, Π_common) then break 16 else if S_missing = ∅ then θ_inc ← θ_new; break 17 else N ← 2 · N 18 if time spent in this call to this procedure exceeds t_intensify and i ≥ 2 then break 19 return [R, θ_inc]

stats: Stats

stats object

traj_logger: TrajLogger

TrajLogger object to log all new incumbents

rng : np.random.RandomState instances : typing.List[str]

list of all instance ids

instance_specificstyping.Mapping[str,np.ndarray]

mapping from instance name to instance specific string

cutoffint

runtime cutoff of TA runs

deterministic: bool

whether the TA is deterministic or not

run_obj_time: bool

whether the run objective is runtime or not (if true, apply adaptive capping)

always_race_against: Configuration

if incumbent changes race this configuration always against new incumbent; can sometimes prevent over-tuning

use_ta_time_bound: bool,

if true, trust time reported by the target algorithms instead of measuring the wallclock time for limiting the time of intensification

run_limitint

Maximum number of target algorithm runs per call to intensify.

maxRint

Maximum number of runs per config (summed over all calls to intensifiy).

minRint

Minimum number of run per config (summed over all calls to intensify).

adaptive_capping_slackfactor: float

slack factor of adpative capping (factor * adpative cutoff)

min_chall: int

minimal number of challengers to be considered (even if time_bound is exhausted earlier)

_generate_challengers(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser]) → Iterator[Optional[ConfigSpace.configuration_space.Configuration]][source]

Retuns a sequence of challengers to use in intensification If challengers are not provided, then optimizer will be used to generate the challenger list

Parameters
Returns

A generator containing the next challengers to use

Return type

typing.Optional[typing.Generator[Configuration]]

_get_inc_available_inst(incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) → List[str][source]

Implementation of line 4 of Intensification

This method queries the inc runs in the run history and return the pending instances if any is available

Parameters
  • incumbent (Configuration) – Either challenger or incumbent

  • run_history (RunHistory) – stores all runs we ran so far

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

_get_instances_to_run(challenger: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, N: int, run_history: smac.runhistory.runhistory.RunHistory) → Tuple[List[smac.runhistory.runhistory.InstSeedBudgetKey], float][source]

Returns the minimum list of <instance, seed> pairs to run the challenger on before comparing it with the incumbent

Parameters
  • incumbent (Configuration) – incumbent configuration

  • challenger (Configuration) – promising configuration that is presently being evaluated

  • run_history (RunHistory) – Stores all runs we ran so far

  • N (int) – number of <instance, seed> pairs to select

Returns

  • typing.List[InstSeedBudgetKey] – list of <instance, seed, budget> tuples to run

  • float – total (runtime) cost of running the incumbent on the instances (used for adaptive capping while racing)

_get_next_inc_run(available_insts: List[str]) → Tuple[str, int, Optional[float]][source]

Method to extract the next seed/instance in which a incumbent run most be evaluated.

Parameters

available_insts (typing.List[str]) – A list of instances from which to extract the next incumbent run

Returns

  • instance (str) – Next instance to evaluate

  • seed (float) – Seed in which to evaluate the instance

  • cutoff (Optional[float]) – Max time for a given instance/seed pair

_get_next_racer(challenger: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) → Tuple[ConfigSpace.configuration_space.Configuration, str, int, Optional[float]][source]

Method to return the next config setting to aggressively race challenger against incumbent.

Parameters
  • challenger (Configuration) – Configuration which challenges incumbent

  • incumbent (Configuration) – Best configuration so far

  • run_history (RunHistory) – Stores all runs we ran so far

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

Returns

  • new_incumbent (Configuration) – Either challenger or incumbent

  • instance (str) – Next instance to evaluate

  • seed (int) – Seed in which to evaluate the instance

  • cutoff (Optional[float]) – Max time for a given instance/seed pair

_is_there_time_due_to_adaptive_cap(challenger: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory) → bool[source]

A check to see if there is no more time for a challenger given the fact, that we are optimizing time and the incumbent looks more promising Line 18

Parameters
  • challenger (Configuration) – Configuration which challenges incumbent

  • run_history (RunHistory) – Stores all runs we ran so far

Returns

whether or not there is more time for a challenger run

Return type

bool

_next_iteration() → None[source]

Updates tracking variables at the end of an intensification run

_process_inc_run(incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) → None[source]

Method to process the results of a challenger that races an incumbent

Parameters
  • incumbent (Configuration) – Either challenger or incumbent

  • run_history (RunHistory) – stores all runs we ran so far

  • log_traj (bool) – Whether to log changes of incumbents in trajectory

_process_racer_results(challenger: ConfigSpace.configuration_space.Configuration, incumbent: ConfigSpace.configuration_space.Configuration, run_history: smac.runhistory.runhistory.RunHistory, log_traj: bool = True) → Optional[ConfigSpace.configuration_space.Configuration][source]

Process the result of a racing configuration against the current incumbent. Might propose a new incumbent.

Parameters
  • challenger (Configuration) – Configuration which challenges incumbent

  • incumbent (Configuration) – Best configuration so far

  • run_history (RunHistory) – Stores all runs we ran so far

Returns

new_incumbent – Either challenger or incumbent

Return type

typing.Optional[Configuration]

get_next_challenger(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser]) → Tuple[Optional[ConfigSpace.configuration_space.Configuration], bool][source]

This function returns the next challenger, that should be exercised though lines 8-17.

It does so by populating configs_to_run, which is a pool of configuration from which the racer will sample. Each configuration within configs_to_run, will be intensified on different instances/seed registered in self.to_run as stated in line 11.

A brand new configuration should only be sampled, after all self.to_run instance seed pairs are exhausted.

This method triggers a call to _next_iteration if there are no more configurations to run, for the current intensification loop. This marks the transition to Line 2, where a new configuration to intensify will be drawn from epm/initial challengers.

Parameters
Returns

  • typing.Optional[Configuration] – next configuration to evaluate

  • bool – flag telling if the configuration is newly sampled or one currently being tracked

get_next_run(challengers: Optional[List[ConfigSpace.configuration_space.Configuration]], incumbent: ConfigSpace.configuration_space.Configuration, chooser: Optional[smac.optimizer.epm_configuration_chooser.EPMChooser], run_history: smac.runhistory.runhistory.RunHistory, repeat_configs: bool = True, num_workers: int = 1) → Tuple[smac.intensification.abstract_racer.RunInfoIntent, smac.runhistory.runhistory.RunInfo][source]

This procedure is in charge of generating a RunInfo object to comply with lines 7 (in case stage is stage==RUN_INCUMBENT) or line 12 (In case of stage==RUN_CHALLENGER)

A RunInfo object encapsulates the necessary information for a worker to execute the job, nevertheless, a challenger is not always available. This could happen because no more configurations are available or the new configuration to try was already executed.

To circumvent this, a intent is also returned:

  • (intent=RUN) Run the RunInfo object (Normal Execution

  • (intent=SKIP) Skip this iteration. No challenger is available, in particular

    because challenger is the same as incumbent

Parameters
  • challengers (typing.List[Configuration]) – promising configurations

  • incumbent (Configuration) – incumbent configuration

  • chooser (smac.optimizer.epm_configuration_chooser.EPMChooser) – optimizer that generates next configurations to use for racing

  • run_history (RunHistory) – stores all runs we ran so far

  • repeat_configs (bool) – if False, an evaluated configuration will not be generated again

  • num_workers (int) – the maximum number of workers available at a given time.

Returns

  • intent (RunInfoIntent) – What should the smbo object do with the runinfo.

  • run_info (RunInfo) – An object that encapsulates necessary information for a config run

process_results(run_info: smac.runhistory.runhistory.RunInfo, incumbent: Optional[ConfigSpace.configuration_space.Configuration], run_history: smac.runhistory.runhistory.RunHistory, time_bound: float, result: smac.runhistory.runhistory.RunValue, log_traj: bool = True) → Tuple[ConfigSpace.configuration_space.Configuration, float][source]

The intensifier stage will be updated based on the results/status of a configuration execution.

During intensification, the following can happen: -> Challenger raced against incumbent -> Also, during a challenger run, a capped exception

can be triggered, where no racer post processing is needed

-> A run on the incumbent for more confidence needs to

be processed, IntensifierStage.PROCESS_INCUMBENT_RUN

-> The first run results need to be processed

(PROCESS_FIRST_CONFIG_RUN)

At the end of any run, checks are done to move to a new iteration.

Parameters
  • run_info (RunInfo) – A RunInfo containing the configuration that was evaluated

  • incumbent (typing.Optional[Configuration]) – best configuration so far, None in 1st run

  • run_history (RunHistory) – stores all runs we ran so far if False, an evaluated configuration will not be generated again

  • time_bound (float) – time in [sec] available to perform intensify

  • result (RunValue) – Contain the result (status and other methadata) of exercising a challenger/incumbent.

  • log_traj (bool) – whether to log changes of incumbents in trajectory

Returns

  • incumbent (Configuration()) – current (maybe new) incumbent configuration

  • inc_perf (float) – empirical performance of incumbent configuration

class smac.intensification.intensification.IntensifierStage(value)[source]

Bases: enum.Enum

Class to define different stages of intensifier

PROCESS_FIRST_CONFIG_RUN = 4
PROCESS_INCUMBENT_RUN = 5
RUN_BASIS = 3
RUN_CHALLENGER = 2
RUN_FIRST_CONFIG = 0
RUN_INCUMBENT = 1
exception smac.intensification.intensification.NoMoreChallengers[source]

Bases: Exception

Indicates that no more challengers are available for the intensification to proceed