Skip to content

Setup#

Installation#

Using pip#

pip install mf-prior-bench

To install specific benchmark dependancies, you can specify any or all of the following:

pip install "mf-prior-bench[yahpo, jahs-bench, pd1]"

From source#

git clone git@github.com:automl/mf-prior-bench

pip install "mf-prior-bench"

To install specific benchmark dependancies, you can specify any or all of the following:

git clone git@github.com:automl/mf-prior-bench

pip install "mf-prior-bench[yahpo, jahs-bench, pd1]"

CLI#

mfpbench provides some helpful commands for downloading data as well as generating priors for benchmarks.

CLI Help
python -m mfpbench --help

usage: python -m mfpbench [-h] {install,download,priors} ...

positional arguments:
  {install,download,priors}
    install             python -m mfpbench install --help

                        Install requirements for use with mfpbench.
                        -------------------------------------------
    download            python -m mfpbench download --help

                        Download datasets for use with mfpbench.
                        ----------------------------------------
    priors              python -m mfpbench priors --help

                        Generate priors for use with mfpbench.
                        --------------------------------------

options:
  -h, --help            show this help message and exit

Download#

Some benchmark require raw files to work, you can download these using the python -m mfpbench download command.

You can specify the location of the root data directory with --data-dir. We recommend leaving this to the default to make working with benchmarks easier.

# Disabled in docs
python -m mfpbench download --benchmark "pd1"

Download Status
python -m mfpbench download --list

yahpo
jahs
pd1
lcbench-tabular

Setup Status
python -m mfpbench download --status

root: /home/runner/work/mf-prior-bench/mf-prior-bench/data
----------------------------------------------------------
[x] yahpo                python -m mfpbench download --benchmark yahpo
[x] jahs                 python -m mfpbench download --benchmark jahs
[x] pd1                  python -m mfpbench download --benchmark pd1
[x] lcbench-tabular      python -m mfpbench download --benchmark lcbench-tabular


# Execution disabled for docs
python -m mfpbench download --benchmark "pd1" --force

Download Help
python -m mfpbench download --help

usage: python -m mfpbench download [-h] [--force] [--status] [--list]
                                   [--benchmark {yahpo,jahs,pd1,lcbench-tabular}]
                                   [--data-dir DATA_DIR]

options:
  -h, --help            show this help message and exit
  --force               Force download and remove existing data
  --status              Print out the status of benchmarks
  --list                Print out the available benchmarks data sources
  --benchmark {yahpo,jahs,pd1,lcbench-tabular}
                        The benchmark to download.
  --data-dir DATA_DIR   Where to save the data, defaults to './data'

Install#

As there will be conflicting dependancies between different benchmarks, we added some requirements files to specifically run these benchmarks. You will still have to setup an environment however you would do so but you can quickly install the required dependancies using the python -m mfpbench install subcommand.

# Disabled in docs
python -m mfpbench install --benchmark "lcbench-tabular"

Install list
python -m mfpbench install --list

yahpo
jahs
pd1
lcbench-tabular

Install view
python -m mfpbench install --view --benchmark "yahpo"

=====
yahpo
=====
path: /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requirements/yahpo.txt
exec: /opt/hostedtoolcache/Python/3.11.6/x64/bin/python
cmd: python -m pip install -r /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requirements/yahpo.txt
------------------------------------------------------------------------------------------------------------------------
# /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/requirements/yahpo.txt
tqdm


# Disabled for docs
python -m mfpbench install --requirements "/path/to/myreqs"

Install help
python -m mfpbench install --help

usage: python -m mfpbench install [-h] [--list] [--view]
                                  [--benchmark {yahpo,jahs,pd1,lcbench-tabular}]
                                  [--requirements REQUIREMENTS]

options:
  -h, --help            show this help message and exit
  --list                Print out the available benchmarks data sources
  --view                View the default requirements for a benchmark
  --benchmark {yahpo,jahs,pd1,lcbench-tabular}
                        The benchmark to setup.
  --requirements REQUIREMENTS
                        The requirements into the currently active
                        environment. If not specified, will use the default
                        requirements. If string, the string will be used as
                        the path to the requirements file.

Generating Priors#

TODO