During optimization, SMAC does not automatically evaluate all configurations on all instances.
validate-module provides a convenient
way to do this after a configuration run is finished. You can specify a set of configurations and a set of instances,
whereupon the validator will run every configuration on every instance. This is
either done by calling the target algorithm or it is estimated using an EPM.
This functionality can be important when comparing default and incumbent, giving
a best estimate of the actual quality of the incumbent or investigating the
quality of the incumbents founds over the time of the optimization.
Validation supports parallel computing, non-deterministic target algorithms and
reusing results from runhistories.
When evaluating the cost via target-algorithm runs (no EPM), runs should be reused from a given runhistory only on comparable hardware!
To validate directly in Python (e.g. to perform a validation immediately after an
Validator can be used. It provides two different
validate_epm. Both return runhistories
containing results for all desired config/instance-pairs.
To validate an actual
it has its own method
validate within SMAC.
python3 smac-validate.py --scenario SCENARIO --trajectory TRAJECTORY --output OUTPUT [--configs CONFIG_MODE] [--instances INSTANCE_MODE] [--[no-]epm] [--runhistory RUNHISTORY] [--seed SEED] [--repetitions REPETITIONS] [--n_jobs N_JOBS] [--tae TAE]
Path to the file that specifies the scenario that is used for the validation.
Path to the trajectory of the SMAC run.
Path to save output-runhistory to.
Which instances to validate on, train or test are defined in scenario. From: [train, test, train+test]. Default: test
What configurations to validate. From: [def, inc, def+inc, wallclock_time, cpu_time, all]. def and inc validate on default and/or incumbent. all validates on all configurations in the specified trajectory. wallclock_time/cpu_time validate at cpu- or wallclock-time steps of [max_time/2^0, max_time/2^1, max_time/2^3, …, default]. Default: def+inc
epm uses an EPM which is built upon the given runhistory to estimate the costs of the config/instance-pairs. no-epm evaluates the config/instance-pairs by actually running the target algorithm. Default: no-epm
Path to a runhistory. If specified, runs from the runhistory will not be re-evaluated. Default: None
For non-deterministic target algorithms, this option specifies the number of different seeds that are evaluated per config/instance-pair. Default: 1.
if no-epm is used, this is the number of cores to use for evaluation in parallel. Default: 1.
From [old, aclib]. If no-epm is used, this specifies the format of the (see TAE). Default: old.
In [INFO, DEBUG]. Specifies the logging-verbosity. Default: INFO.
Seed to be used for validation