`mfpbench`#

Welcome to the documentation of the Multi-Fidelity Priors Benchmark library. Check out the quickstart to get started or check out our examples.

At a glance, this library wraps several benchmarks that give access to the multi-fidelity, learning curve information of configurations from a static configuration space, as well as allow you to set the default configuration of some search space as the prior. This mainly includes validation score/error as well as cost.

This includes any necessary setup commands which you can find in the setup.

Terminology? Deep Learning Example

In the deep learning setting, we often train configurations of a Neural Network for several epochs, recording the performance as it goes. However, finding the best configurations requires searching over some possible set Every configuration is going to take a different amount of time to train but often you'd like to find the best configuration as fast as possible.

You start with your intutition and choose learning_rate=0.001 and apply some augmentation=True, however when searching you also allow different values. You split your data into "train", "val", "test" sets, and set up your pipeline so your network trains on "train", you score how good the hyperparameters are on "val" to choose the best but you also evaluate these configurations on "test". You do this last part to get a sense of how much you are overfitting.

fidelity - Here this would be the epochs
configuration - One particular Neural Network architecture and it's hyperparameters.
configuration space - The space of possible values for all the hyperparameters, or in some cases, a finit set of distinct configurations.
prior - The default configuration of learning_rate=0.001 and augmentation=True.
validation score - How well your model does on the "val" set.
cost - The amount of time it took to train a model
learning curve - The ordered sequence of results for one configuration for each fidelity value.

Example

import mfpbench

benchmark = mfpbench.get("mfh3", bias=2.5, noise=0)

config = benchmark.sample()
result = benchmark.query(config, at=34)

trajectory = benchmark.trajectory(config, frm=5, to=benchmark.end)

print(config)
print(result)
print(len(trajectory))

MFHartmann3Config(X_0=0.7343471694568163, X_1=0.9433106688641351, X_2=0.821513935537763)
MFHartmannResult(fidelity=34, value=-0.6829846406575995, fid_cost=0.15982000000000002)
96

Benchmarks#

pd1: A set of optimization results for larger deep-learning models from the HyperBO paper. We use this raw tabular data and build surrogate xgboost models to provide a smooth continuious space.
yahpo-gym A collection of surrogate models trained from evaluations of various models and search spaces over a variety of models and tasks.
MFHartmann: A synthetic benchmark for multifidelity optimization that allows you to control the noise and the crossing of learning curves. Useful for testing the capabilities of optimizers.
LCBenchTabular: A set of tabular benchmarks from LCBench, which train various MLP's with AutoPyTorch on different OpenML tasks.
Your own: There are also options to easily integrate your own benchmarks, whether from raw tables or with a more sophisticated objective function.

mfpbench#

Benchmarks#

`mfpbench`#