Input Format

Supported file-formats for configurator-output are currently the output of SMAC2, SMAC3 and BOHB, as well as csv-data formatted as specified below.

SMAC2

Relevant files for the analysis of SMAC2 (relative to the specified folder with ??? as wildcards for digits) are:

  • scenario.txt
  • run_and_results-it???.csv
  • paramstrings-it???.txt
  • ../traj-run-?.txt

plus the files specified in the scenario.txt (pcs_fn.pcs, instance- and instancefeature-fn.txt, …)

SMAC3

Relevant files for the analysis of SMAC3 (relative to the specified folder) are:

  • scenario.txt
  • runhistory.json
  • traj_aclib2.json
  • validated_runhistory.json (optional)

plus the files specified in the scenario.txt (pcs_fn.pcs, instance- and instancefeature-fn.txt, …)

BOHB

To analyzer BOHB-results, you need to run BOHB with a result-logger (check here for an example on how to do it). The folder you specify must contain:

  • configs.json
  • results.json
  • configspace.pcs

configs.json and results.json will be automatically logged by the result-logger. Save a ConfigSpace config_space in a directory output_dir using

from ConfigSpace.read_and_write import pcs_new
with open(output_dir, 'w') as fh:
    fh.write(pcs_new(config_space))

CSV

Comma-separated values give the opportunity to parse all the rundata in a very simple format. A scenario-file (and therein specified files) is still required. runhistory.csv substitutes the runhistory, each line representing one target algorithm run. The first line of the file is the header, with the following entries: cost, time, status, seed, config_id and instance_id. When specifying config_id, the specified folder must also contain a configurations.csv. If you use instance_id-columns, the instance_ids must be the same that are specified in the instance-feature-file.

instance_features.csv (path for instance_features is used from provided scenario):

INSTANCE_ID inst_feat_name1 inst_feat_name2
0 value1 value2

configurations.csv:

CONFIG_ID parameter_name1 parameter_name2
0 value1 value2

runhistory.csv:

config_id instance_id cost time seed status
name of config 1 name of instance 1

trajectory.csv:

cpu_time cost wallclock_time incumbent

Alternatively CAVE can also read in one runhistory.csv-file containing all the information about parameters and instances, in this case the file configurations.csv is not needed. See below for example:

runhistory.csv:

cost time seed status parameter1 parameter2 inst_feat1 inst_feat2

trajectory.csv:

cpu_time cost wallclock_time parameter1 parameter2