smac.utils.validate module

class smac.utils.validate.Validator(scenario: smac.scenario.scenario.Scenario, trajectory: Optional[List], rng: Optional[Union[numpy.random.mtrand.RandomState, int]] = None)[source]

Bases: object

Validator for the output of SMAC-scenarios. Evaluates specified configurations on specified instances.

Construct Validator for given scenario and trajectory.

Parameters
  • scenario (Scenario) – scenario object for cutoff, instances, features and specifics

  • trajectory (trajectory-list) – trajectory to take incumbent(s) from

  • rng (np.random.RandomState or int) – Random number generator or seed

_get_configs(mode: str) → List[str][source]

Return desired configs

Parameters

mode (str) –

str from [def, inc, def+inc, wallclock_time, cpu_time, all]

time evaluates at cpu- or wallclock-timesteps of: [max_time/2^0, max_time/2^1, max_time/2^3, …, default] with max_time being the highest recorded time

Returns

configs – list with desired configurations

Return type

list<Configuration>

_get_instances(mode: str) → List[str][source]

Get desired instances

Parameters

mode (str) – what instances to use for validation, from [train, test, train+test]

Returns

instances – instances to be used

Return type

list<str>

_get_runs(configs: Union[str, List[ConfigSpace.configuration_space.Configuration]], insts: Union[str, List[str]], repetitions: int = 1, runhistory: smac.runhistory.runhistory.RunHistory = None) → Tuple[List[smac.utils.validate.Run], smac.runhistory.runhistory.RunHistory][source]

Generate list of SMAC-TAE runs to be executed. This means combinations of configs with all instances on a certain number of seeds.

side effect: Adds runs that don’t need to be reevaluated to self.rh!

Parameters
  • configs (str or list<Configuration>) –

    string or directly a list of Configuration str from [def, inc, def+inc, wallclock_time, cpu_time, all]

    time evaluates at cpu- or wallclock-timesteps of: [max_time/2^0, max_time/2^1, max_time/2^3, …, default] with max_time being the highest recorded time

  • insts (str or list<str>) – what instances to use for validation, either from [train, test, train+test] or directly a list of instances

  • repetitions (int) – number of seeds per instance/config-pair to be evaluated

  • runhistory (RunHistory) – optional, try to reuse this runhistory and save some runs

Returns

runs – list with _Runs [_Run(config=CONFIG1,inst=INSTANCE1,seed=SEED1,inst_specs=INST_SPECIFICS1),

_Run(config=CONFIG2,inst=INSTANCE2,seed=SEED2,inst_specs=INST_SPECIFICS2), …]

Return type

list<_Run>

_process_runhistory(configs: List[ConfigSpace.configuration_space.Configuration], insts: Sequence[Optional[str]], runhistory: Optional[smac.runhistory.runhistory.RunHistory]) → Dict[str, List[Tuple[int, List[ConfigSpace.configuration_space.Configuration]]]][source]

Processes runhistory from self._get_runs by extracting already evaluated (relevant) config-inst-seed tuples.

Parameters
  • configs (list(Configuration)) – list of configs of interest

  • insts (list(str)) – list of instances of interest

  • runhistory (RunHistory) – runhistory to extract runs from

Returns

inst_seed_config – dictionary mapping instances to a list of tuples of already used seeds and the configs that this inst-seed-pair has been evaluated on, sorted by the number of configs

Return type

dict<str : list(tuple(int, tuple(configs)))>

_save_results(rh: smac.runhistory.runhistory.RunHistory, output_fn: Optional[str], backup_fn: Optional[str] = None) → None[source]

Helper to save results to file

Parameters
  • rh (RunHistory) – runhistory to save

  • output_fn (str) – if ends on ‘.json’: filename to save history to else: directory to save runhistory to (filename is backup_fn)

  • backup_fn (str) – if output_fn does not end on ‘.json’, treat output_fn as dir and append backup_fn as filename (if output_fn ends on ‘.json’, this argument is ignored)

_validate_parallel(tae: smac.tae.execute_ta_run.ExecuteTARun, runs: List[smac.utils.validate.Run], n_jobs: int, backend: str) → List[Tuple[smac.tae.execute_ta_run.StatusType, float, float, Dict]][source]

Validate runs with joblibs Parallel-interface

Parameters
  • tae (ExecuteTARun) – tae to be used for validation

  • runs (list<_Run>) – list with _Run-objects [_Run(config=CONFIG1,inst=INSTANCE1,seed=SEED1,inst_specs=INST_SPECIFICS1), …]

  • n_jobs (int) – number of cpus to use for validation (-1 to use all)

  • backend (str) – what backend to use for parallelization

Returns

run_results – results as returned by tae

Return type

list<tuple(tae-returns)>

validate(config_mode: Union[str, List[ConfigSpace.configuration_space.Configuration]] = 'def', instance_mode: Union[str, List[str]] = 'test', repetitions: int = 1, n_jobs: int = 1, backend: str = 'threading', runhistory: smac.runhistory.runhistory.RunHistory = None, tae: smac.tae.execute_ta_run.ExecuteTARun = None, output_fn: Optional[str] = None)smac.runhistory.runhistory.RunHistory[source]

Validate configs on instances and save result in runhistory. If a runhistory is provided as input it is important that you run it on the same/comparable hardware.

side effect: if output is specified, saves runhistory to specified output directory.

Parameters
  • config_mode (str or list<Configuration>) – string or directly a list of Configuration. string from [def, inc, def+inc, wallclock_time, cpu_time, all]. time evaluates at cpu- or wallclock-timesteps of: [max_time/2^0, max_time/2^1, max_time/2^3, …, default] with max_time being the highest recorded time

  • instance_mode (str or list<str>) – what instances to use for validation, either from [train, test, train+test] or directly a list of instances

  • repetitions (int) – number of repetitions in nondeterministic algorithms

  • n_jobs (int) – number of parallel processes used by joblib

  • backend (str) – what backend joblib should use for parallel runs

  • runhistory (RunHistory) – optional, RunHistory-object to reuse runs

  • tae (ExecuteTARun) – tae to be used. if None, will initialize ExecuteTARunOld

  • output_fn (str) – path to runhistory to be saved. if the suffix is not ‘.json’, will be interpreted as directory and filename will be ‘validated_runhistory.json’

Returns

runhistory – runhistory with validated runs

Return type

RunHistory

validate_epm(config_mode: Union[str, List[ConfigSpace.configuration_space.Configuration]] = 'def', instance_mode: Union[str, List[str]] = 'test', repetitions: int = 1, runhistory: Optional[smac.runhistory.runhistory.RunHistory] = None, output_fn: Optional[str] = None, reuse_epm: bool = True)smac.runhistory.runhistory.RunHistory[source]

Use EPM to predict costs/runtimes for unknown config/inst-pairs.

side effect: if output is specified, saves runhistory to specified output directory.

Parameters
  • output_fn (str) – path to runhistory to be saved. if the suffix is not ‘.json’, will be interpreted as directory and filename will be ‘validated_runhistory_EPM.json’

  • config_mode (str or list<Configuration>) – string or directly a list of Configuration, string from [def, inc, def+inc, wallclock_time, cpu_time, all]. time evaluates at cpu- or wallclock-timesteps of: [max_time/2^0, max_time/2^1, max_time/2^3, …, default] with max_time being the highest recorded time

  • instance_mode (str or list<str>) – what instances to use for validation, either from [train, test, train+test] or directly a list of instances

  • repetitions (int) – number of repetitions in nondeterministic algorithms

  • runhistory (RunHistory) – optional, RunHistory-object to reuse runs

  • reuse_epm (bool) – if true (and if self.epm), reuse epm to validate runs

Returns

runhistory – runhistory with predicted runs

Return type

RunHistory

smac.utils.validate._Run

alias of smac.utils.validate.Run

smac.utils.validate._unbound_tae_starter(tae: smac.tae.execute_ta_run.ExecuteTARun, *args: Any, **kwargs: Any) → Tuple[smac.tae.execute_ta_run.StatusType, float, float, Dict][source]

Unbound function to be used by joblibs Parallel, since directly passing the TAE results in pickling-problems.

Parameters
  • tae (ExecuteTARun) – tae to be used

  • *args (various) – arguments to the tae

  • **kwargs (various) – arguments to the tae

Returns

tae_results – return from tae.start

Return type

tuple