smac.utils.validate

Classes

Validator(scenario, trajectory[, rng])

Validator for the output of SMAC-scenarios.
class smac.utils.validate.Validator(scenario, trajectory, rng=None)[source]

Bases: object

Validator for the output of SMAC-scenarios.

Evaluates specified configurations on specified instances.

Parameters

  • scenario (Scenario) – scenario object for cutoff, instances, features and specifics

  • trajectory (trajectory-list) – trajectory to take incumbent(s) from

  • rng (np.random.RandomState or int) – Random number generator or seed

validate(config_mode='def', instance_mode='test', repetitions=1, n_jobs=1, backend='threading', runhistory=None, tae=None, output_fn=None)[source]

Validate configs on instances and save result in runhistory. If a runhistory is provided as input it is important that you run it on the same/comparable hardware.

side effect: if output is specified, saves runhistory to specified output directory.

Parameters

  • config_mode (str or list<Configuration>) – string or directly a list of Configuration. string from [def, inc, def+inc, wallclock_time, cpu_time, all]. time evaluates at cpu- or wallclock-timesteps of: [max_time/2^0, max_time/2^1, max_time/2^3, …, default] with max_time being the highest recorded time

  • instance_mode (str or list<str>) – what instances to use for validation, either from [train, test, train+test] or directly a list of instances

  • repetitions (int) – number of repetitions in nondeterministic algorithms

  • n_jobs (int) – number of parallel processes used by joblib

  • backend (str) – what backend joblib should use for parallel runs

  • runhistory (RunHistory) – optional, RunHistory-object to reuse runs

  • tae (BaseRunner) – tae to be used. if None, will initialize ExecuteTARunOld

  • output_fn (str) – path to runhistory to be saved. if the suffix is not ‘.json’, will be interpreted as directory and filename will be ‘validated_runhistory.json’

Returns

runhistory – runhistory with validated runs

Return type

RunHistory

validate_epm(config_mode='def', instance_mode='test', repetitions=1, runhistory=None, output_fn=None, reuse_epm=True)[source]

Use EPM to predict costs/runtimes for unknown config/inst-pairs.

side effect: if output is specified, saves runhistory to specified output directory.

Parameters

  • output_fn (str) – path to runhistory to be saved. if the suffix is not ‘.json’, will be interpreted as directory and filename will be ‘validated_runhistory_EPM.json’

  • config_mode (str or list<Configuration>) – string or directly a list of Configuration, string from [def, inc, def+inc, wallclock_time, cpu_time, all]. time evaluates at cpu- or wallclock-timesteps of: [max_time/2^0, max_time/2^1, max_time/2^3, …, default] with max_time being the highest recorded time

  • instance_mode (str or list<str>) – what instances to use for validation, either from [train, test, train+test] or directly a list of instances

  • repetitions (int) – number of repetitions in nondeterministic algorithms

  • runhistory (RunHistory) – optional, RunHistory-object to reuse runs

  • reuse_epm (bool) – if true (and if self.epm), reuse epm to validate runs

Returns

runhistory – runhistory with predicted runs

Return type

RunHistory

previous

smac.utils.test_helpers

next

Glossary