CARP-S

Welcome to the CARP-S documentation.

Tip

See the navigation links in the header or side-bars. Click the button (top left) on mobile.

For information on how to install CARP-S, please refer to the installation instructions. To see an example of how to run a first optimizer on a benchmark, as well as a more comprehensive overview of what CARP-S can offer, please refer to our guides. If you are interested in the commands that CARP-S provides, please refer to the commands. To learn how to add your own optimizer or benchmark, or if you want to contribute to CARP-S in general, check out our contributing section.

What is CARP-S?

CARP-S is a benchmarking framework for Comprehensive Automated Research Performance Studies, designed to evaluate multiple optimizers across various benchmark tasks. It aims to facilitate the development of effective and reliable methods for hyperparameter optimization (HPO).

• Develop With Confidence

  Developing optimizers is hard, evaluating them is harder and benchmarking against state-of-the-art is even harder. CARP-S provides a simple and easily extendable framework to develop, evaluate and benchmark your optimizer.

• Various Scenarios

  The benchmarks integrated in CARP-S offer tasks for blackbox, multi-fidelity, multi-objective and multi-fidelity-multi-objective HPO, allowing you to evaluate your optimizer in a variety of scenarios.

• Effective Evaluation

  Fair comparisons between optimizers require evaluation on a large number of tasks, which can be computationally infeasible. CARP-S provides a subset of representative benchmarking tasks for each scenario to allow for effective evaluation.

• Minimal Dependencies

  CARP-S is modular and allows you to only install what you need to run and evaluate what you want. For details, please have a look at the installation instructions.