Skip to content

Commands#

Default#

Run combo of task and optimizer#

You can run a certain task and optimizer combination directly with Hydra via: 

python -m carps.run +task=... +optimizer=... seed=... -m

Check missing runs#

To check whether any runs are missing, you can use the following command. It will create a file runcommands_missing.sh containing the missing runs: 

python -m carps.utils.check_missing <rundir>

Gather data of runs#

To collect all run data generated by the file logger into parquet files, use the following command: 

python -m carps.analysis.gather_data <rundir>
The parquet files are then located in <rundir>. The logs.csv contain the trial info and values and the logs_cfg.parquet contain the experiment configuration. The experiments can be matched via the column experiment_id.

Database#

Fill the database with tasks#

Another option is to fill the database with all possible combinations of tasks and optimizers you would like to run: 

python -m carps.container.create_cluster_configs +task=... +optimizer=... -m

Run from database#

Then, run them from the database with: 

python -m carps.run_from_db

Show database statistics#

python -m carps.experimenter.database.show_stats

Reset experiments#

Experiments with error status (or any other status) can be reset via: - python -m carps.experimenter.database.reset_experiments for resetting all experiments with status error - python -m carps.experimenter.database.reset_experiments ['yahpo_attr_error'] for resetting experiments with this specific yahpo error condition. - python -m carps.experimenter.database.reset_experiments ['falsely_done'] for resetting experiments that are falsely marked as done (experiment ids not present in trials table). - python -m carps.experimenter.database.reset_experiments ['error', 'yahpo_attr_error', 'falsely_done'] for resetting all.

Download results from database#

python -m carps.experimenter.database.download_results

Postprocess results from database#

python -m carps.experimenter.database.process_logs

Filelogs to database#

If you ran experiments which logged to files, you can add them post-hoc to the database with the following command: 

python -m carps.experimenter.scrape_results_to_db <rundir1> <rundir2>