Quadratic Function

An example of applying SMAC to optimize a quadratic function.

We use the black-box facade because it is designed for black-box function optimization. The black-box facade uses a Gaussian Process as its surrogate model. The facade works best on a numerical hyperparameter configuration space and should not be applied to problems with large evaluation budgets (up to 1000 evaluations).

[INFO][abstract_initial_design.py:147] Using 10 initial design configurations and 0 additional configurations.
[INFO][abstract_intensifier.py:305] Using only one seed for deterministic scenario.
[INFO][abstract_intensifier.py:515] Added config 3ce33d as new incumbent because there are no incumbents yet.
[INFO][abstract_intensifier.py:590] Added config c53710 and rejected config 3ce33d as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -2.331214789301157 -> 2.1608731895685196
[INFO][abstract_intensifier.py:590] Added config 50b580 and rejected config c53710 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: 2.1608731895685196 -> 0.4316552169620991
[INFO][abstract_intensifier.py:590] Added config d16e41 and rejected config 50b580 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: 0.4316552169620991 -> -0.3100988641381264
[INFO][abstract_intensifier.py:590] Added config 4dcc97 and rejected config d16e41 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.3100988641381264 -> -0.22015188896172422
[INFO][abstract_intensifier.py:590] Added config 4d21be and rejected config 4dcc97 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.22015188896172422 -> -0.1481212818284119
[INFO][abstract_intensifier.py:590] Added config 348898 and rejected config 4d21be as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.1481212818284119 -> -0.13225703373856135
[INFO][abstract_intensifier.py:590] Added config c56201 and rejected config 348898 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.13225703373856135 -> -0.008886632260822758
[INFO][abstract_intensifier.py:590] Added config b6eee0 and rejected config c56201 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.008886632260822758 -> 0.00353744136744627
[INFO][abstract_intensifier.py:590] Added config 06e28a and rejected config b6eee0 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: 0.00353744136744627 -> 0.0029625967410442655
[INFO][abstract_intensifier.py:590] Added config 4026d1 and rejected config 06e28a as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: 0.0029625967410442655 -> 0.0009114180407276962
[INFO][abstract_intensifier.py:590] Added config 025ea5 and rejected config 4026d1 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: 0.0009114180407276962 -> -0.00032238773647641494
[INFO][smbo.py:299] Finished 50 trials.
[INFO][abstract_intensifier.py:590] Added config 342683 and rejected config 025ea5 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.00032238773647641494 -> -0.0002551446012351022
[INFO][abstract_intensifier.py:590] Added config 860384 and rejected config 342683 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][configspace.py:175] --- x: -0.0002551446012351022 -> 6.682757473086554e-05
[INFO][smbo.py:299] Finished 100 trials.
[INFO][smbo.py:307] Configuration budget is exhausted:
[INFO][smbo.py:308] --- Remaining wallclock time: inf
[INFO][smbo.py:309] --- Remaining cpu time: inf
[INFO][smbo.py:310] --- Remaining trials: 0
[INFO][abstract_intensifier.py:305] Using only one seed for deterministic scenario.
Default cost: 25.0
Incumbent cost: 4.465924744409418e-09

import numpy as np
from ConfigSpace import Configuration, ConfigurationSpace, Float
from matplotlib import pyplot as plt

from smac import HyperparameterOptimizationFacade as HPOFacade
from smac import RunHistory, Scenario

__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"


class QuadraticFunction:
    @property
    def configspace(self) -> ConfigurationSpace:
        cs = ConfigurationSpace(seed=0)
        x = Float("x", (-5, 5), default=-5)
        cs.add_hyperparameters([x])

        return cs

    def train(self, config: Configuration, seed: int = 0) -> float:
        """Returns the y value of a quadratic function with a minimum we know to be at x=0."""
        x = config["x"]
        return x**2


def plot(runhistory: RunHistory, incumbent: Configuration) -> None:
    plt.figure()

    # Plot ground truth
    x = list(np.linspace(-5, 5, 100))
    y = [xi * xi for xi in x]
    plt.plot(x, y)

    # Plot all trials
    for k, v in runhistory.items():
        config = runhistory.get_config(k.config_id)
        x = config["x"]
        y = v.cost  # type: ignore
        plt.scatter(x, y, c="blue", alpha=0.1, zorder=9999, marker="o")

    # Plot incumbent
    plt.scatter(incumbent["x"], incumbent["x"] * incumbent["x"], c="red", zorder=10000, marker="x")

    plt.show()


if __name__ == "__main__":
    model = QuadraticFunction()

    # Scenario object specifying the optimization "environment"
    scenario = Scenario(model.configspace, deterministic=True, n_trials=100)

    # Now we use SMAC to find the best hyperparameters
    smac = HPOFacade(
        scenario,
        model.train,  # We pass the target function here
        overwrite=True,  # Overrides any previous results that are found that are inconsistent with the meta-data
    )

    incumbent = smac.optimize()

    # Get cost of default configuration
    default_cost = smac.validate(model.configspace.get_default_configuration())
    print(f"Default cost: {default_cost}")

    # Let's calculate the cost of the incumbent
    incumbent_cost = smac.validate(incumbent)
    print(f"Incumbent cost: {incumbent_cost}")

    # Let's plot it too
    plot(smac.runhistory, incumbent)

previous

Basics

next

Support Vector Machine with Cross-Validation