Quadratic Function

An example of applying SMAC to optimize a quadratic function.

We use the black-box facade because it is designed for black-box function optimization. The black-box facade uses a Gaussian Process as its surrogate model. The facade works best on a numerical hyperparameter configuration space and should not be applied to problems with large evaluation budgets (up to 1000 evaluations).

[INFO][abstract_initial_design.py:147] Using 10 initial design configurations and 0 additional configurations.
[INFO][abstract_intensifier.py:305] Using only one seed for deterministic scenario.
[INFO][abstract_intensifier.py:515] Added config f09c3b as new incumbent because there are no incumbents yet.
[INFO][abstract_intensifier.py:594] Added config bec0fc and rejected config f09c3b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config a34626 and rejected config bec0fc as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config f72805 and rejected config a34626 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config d7ecca and rejected config f72805 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 10bb50 and rejected config d7ecca as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 6a2c79 and rejected config 10bb50 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 4380b1 and rejected config 6a2c79 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 15dfc6 and rejected config 4380b1 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config b9f3bd and rejected config 15dfc6 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config c714f5 and rejected config b9f3bd as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 2da694 and rejected config c714f5 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:319] Finished 50 trials.
[INFO][abstract_intensifier.py:594] Added config e50a55 and rejected config 2da694 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 4a175d and rejected config e50a55 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config fef2f4 and rejected config 4a175d as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 8b3403 and rejected config fef2f4 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config a028a8 and rejected config 8b3403 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config e1dc2b and rejected config a028a8 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config e66a0e and rejected config e1dc2b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 9f5b5b and rejected config e66a0e as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 8d6767 and rejected config 9f5b5b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 1add8f and rejected config 8d6767 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config df326a and rejected config 1add8f as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config 2d8007 and rejected config df326a as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:594] Added config cabaca and rejected config 2d8007 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:319] Finished 100 trials.
[INFO][smbo.py:327] Configuration budget is exhausted:
[INFO][smbo.py:328] --- Remaining wallclock time: inf
[INFO][smbo.py:329] --- Remaining cpu time: inf
[INFO][smbo.py:330] --- Remaining trials: 0
[INFO][abstract_intensifier.py:305] Using only one seed for deterministic scenario.
Default cost: 25.0
Incumbent cost: 1.7011175854063811e-09

import numpy as np
from ConfigSpace import Configuration, ConfigurationSpace, Float
from matplotlib import pyplot as plt

from smac import HyperparameterOptimizationFacade as HPOFacade
from smac import RunHistory, Scenario

__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"


class QuadraticFunction:
    @property
    def configspace(self) -> ConfigurationSpace:
        cs = ConfigurationSpace(seed=0)
        x = Float("x", (-5, 5), default=-5)
        cs.add_hyperparameters([x])

        return cs

    def train(self, config: Configuration, seed: int = 0) -> float:
        """Returns the y value of a quadratic function with a minimum we know to be at x=0."""
        x = config["x"]
        return x**2


def plot(runhistory: RunHistory, incumbent: Configuration) -> None:
    plt.figure()

    # Plot ground truth
    x = list(np.linspace(-5, 5, 100))
    y = [xi * xi for xi in x]
    plt.plot(x, y)

    # Plot all trials
    for k, v in runhistory.items():
        config = runhistory.get_config(k.config_id)
        x = config["x"]
        y = v.cost  # type: ignore
        plt.scatter(x, y, c="blue", alpha=0.1, zorder=9999, marker="o")

    # Plot incumbent
    plt.scatter(incumbent["x"], incumbent["x"] * incumbent["x"], c="red", zorder=10000, marker="x")

    plt.show()


if __name__ == "__main__":
    model = QuadraticFunction()

    # Scenario object specifying the optimization "environment"
    scenario = Scenario(model.configspace, deterministic=True, n_trials=100)

    # Now we use SMAC to find the best hyperparameters
    smac = HPOFacade(
        scenario,
        model.train,  # We pass the target function here
        overwrite=True,  # Overrides any previous results that are found that are inconsistent with the meta-data
    )

    incumbent = smac.optimize()

    # Get cost of default configuration
    default_cost = smac.validate(model.configspace.get_default_configuration())
    print(f"Default cost: {default_cost}")

    # Let's calculate the cost of the incumbent
    incumbent_cost = smac.validate(incumbent)
    print(f"Incumbent cost: {incumbent_cost}")

    # Let's plot it too
    plot(smac.runhistory, incumbent)

previous

Basics

next

Support Vector Machine with Cross-Validation