Ask-and-Tell

This examples show how to use the Ask-and-Tell interface.

[INFO][abstract_initial_design.py:147] Using 20 initial design configurations and 0 additional configurations.
[INFO][abstract_intensifier.py:515] Added config 38310a as new incumbent because there are no incumbents yet.
[INFO][abstract_intensifier.py:594] Added config a840bc and rejected config 38310a as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 9a770c and rejected config a840bc as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 6a6aa0 and rejected config 9a770c as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 27e052 and rejected config 6a6aa0 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:319] Finished 50 trials.
[INFO][abstract_intensifier.py:594] Added config da5bdf and rejected config 27e052 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 4e3320 and rejected config da5bdf as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 58c3ff and rejected config 4e3320 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config ad8450 and rejected config 58c3ff as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 53144c and rejected config ad8450 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 1cfe87 and rejected config 53144c as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:319] Finished 100 trials.
[INFO][smbo.py:327] Configuration budget is exhausted:
[INFO][smbo.py:328] --- Remaining wallclock time: inf
[INFO][smbo.py:329] --- Remaining cpu time: inf
[INFO][smbo.py:330] --- Remaining trials: 0
Default cost: 16916.0
Incumbent cost: 0.6764193577937531

from ConfigSpace import Configuration, ConfigurationSpace, Float

from smac import HyperparameterOptimizationFacade, Scenario
from smac.runhistory.dataclasses import TrialValue

__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"


class Rosenbrock2D:
    @property
    def configspace(self) -> ConfigurationSpace:
        cs = ConfigurationSpace(seed=0)
        x0 = Float("x0", (-5, 10), default=-3)
        x1 = Float("x1", (-5, 10), default=-4)
        cs.add_hyperparameters([x0, x1])

        return cs

    def train(self, config: Configuration, seed: int = 0) -> float:
        """The 2-dimensional Rosenbrock function as a toy model.
        The Rosenbrock function is well know in the optimization community and
        often serves as a toy problem. It can be defined for arbitrary
        dimensions. The minimium is always at x_i = 1 with a function value of
        zero. All input parameters are continuous. The search domain for
        all x's is the interval [-5, 10].
        """
        x1 = config["x0"]
        x2 = config["x1"]

        cost = 100.0 * (x2 - x1**2.0) ** 2.0 + (1 - x1) ** 2.0
        return cost


if __name__ == "__main__":
    model = Rosenbrock2D()

    # Scenario object
    scenario = Scenario(model.configspace, deterministic=False, n_trials=100)

    intensifier = HyperparameterOptimizationFacade.get_intensifier(
        scenario,
        max_config_calls=1,  # We basically use one seed per config only
    )

    # Now we use SMAC to find the best hyperparameters
    smac = HyperparameterOptimizationFacade(
        scenario,
        model.train,
        intensifier=intensifier,
        overwrite=True,
    )

    # We can ask SMAC which trials should be evaluated next
    for _ in range(10):
        info = smac.ask()
        assert info.seed is not None

        cost = model.train(info.config, seed=info.seed)
        value = TrialValue(cost=cost, time=0.5)

        smac.tell(info, value)

    # After calling ask+tell, we can still optimize
    # Note: SMAC will optimize the next 90 trials because 10 trials already have been evaluated
    incumbent = smac.optimize()

    # Get cost of default configuration
    default_cost = smac.validate(model.configspace.get_default_configuration())
    print(f"Default cost: {default_cost}")

    # Let's calculate the cost of the incumbent
    incumbent_cost = smac.validate(incumbent)
    print(f"Incumbent cost: {incumbent_cost}")

previous

Support Vector Machine with Cross-Validation

next

Custom Callback