During optimization, SMAC does not automatically evaluate all configurations on all instances. The validate-module provides a convenient way to do this after a configuration run is finished. You can specify a set of configurations and a set of instances, whereupon the validator will run every configuration on every instance. This is either done by calling the target algorithm or it is estimated using an EPM. This functionality can be important when comparing default and incumbent, giving a best estimate of the actual quality of the incumbent or investigating the quality of the incumbents founds over the time of the optimization. Validation supports parallel computing, non-deterministic target algorithms and reusing results from runhistories.


When evaluating the cost via target-algorithm runs (no EPM), runs should be reused from a given runhistory only on comparable hardware!


To validate directly in Python (e.g. to perform a validation immediately after an optimization), the Validator can be used. It provides two different methods, validate and validate_epm. Both return runhistories containing results for all desired config/instance-pairs. To validate an actual SMAC-object, it has its own method validate within SMAC.


python3 --scenario SCENARIO --trajectory TRAJECTORY --output OUTPUT [--configs CONFIG_MODE] [--instances INSTANCE_MODE] [--[no-]epm] [--runhistory RUNHISTORY] [--seed SEED] [--repetitions REPETITIONS] [--n_jobs N_JOBS] [--tae TAE]

Path to the file that specifies the scenario that is used for the validation.


Path to the trajectory of the SMAC run.


Path to save output-runhistory to.


Which instances to validate on, train or test are defined in scenario. From: [train, test, train+test]. Default: test


What configurations to validate. From: [def, inc, def+inc, wallclock_time, cpu_time, all]. def and inc validate on default and/or incumbent. all validates on all configurations in the specified trajectory. wallclock_time/cpu_time validate at cpu- or wallclock-time steps of [max_time/2^0, max_time/2^1, max_time/2^3, …, default]. Default: def+inc


epm uses an EPM which is built upon the given runhistory to estimate the costs of the config/instance-pairs. no-epm evaluates the config/instance-pairs by actually running the target algorithm. Default: no-epm


Path to a runhistory. If specified, runs from the runhistory will not be re-evaluated. Default: None


For non-deterministic target algorithms, this option specifies the number of different seeds that are evaluated per config/instance-pair. Default: 1.


if no-epm is used, this is the number of cores to use for evaluation in parallel. Default: 1.


From [old, aclib]. If no-epm is used, this specifies the format of the (see TAE). Default: old.


In [INFO, DEBUG]. Specifies the logging-verbosity. Default: INFO.


Seed to be used for validation