Specify Number of Trials via a Total Budget in Hyperband

This example uses a dummy function but illustrates how to setup Hyperband if you want to specify a total optimization budget in terms of fidelity units.

In Hyperband, normally SMAC calculates a typical Hyperband round. If the number of trials is not used up by one single round, the next round is started. Instead of specifying the number of trial beforehand, specify the total budget in terms of the fidelity units and let SMAC calculate how many trials that would be.

------------------------------ HYPERBAND IN MULTI-FIDELITY ------------------------------
total budget:            10000
total number of trials:  249
number of HB rounds:     3.802816901408451

        ~~~~~~~~~~~~HYPERBAND ROUND
        eta:                                     3
        min budget per trial:                    10
        max budget per trial:                    500
        total number of trials per HB round:     69
        budget used per HB round:                2629.6296296296296
        number of brackets:                      3
        budgets per stage:                       {0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0], 1: [55.55555555555555, 166.66666666666666, 500.0], 2: [166.66666666666666, 500.0], 3: [500.0]}
        n configs per stage:                     {0: [27, 9, 3, 1], 1: [12, 4, 1], 2: [6, 2], 3: [4]}
-----------------------------------------------------------------------------------------
[INFO][abstract_initial_design.py:147] Using 10 initial design configurations and 0 additional configurations.
[INFO][abstract_intensifier.py:306] Using only one seed for deterministic scenario.
[INFO][successive_halving.py:164] Successive Halving uses budget type BUDGETS with eta 3, min budget 10, and max budget 500.
[INFO][successive_halving.py:323] Number of configs in stage:
[INFO][successive_halving.py:325] --- Bracket 0: [27, 9, 3, 1]
[INFO][successive_halving.py:325] --- Bracket 1: [12, 4, 1]
[INFO][successive_halving.py:325] --- Bracket 2: [6, 2]
[INFO][successive_halving.py:325] --- Bracket 3: [4]
[INFO][successive_halving.py:327] Budgets in stage:
[INFO][successive_halving.py:329] --- Bracket 0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 1: [55.55555555555555, 166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 2: [166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 3: [500.0]
[INFO][abstract_intensifier.py:516] Added config ce0554 as new incumbent because there are no incumbents yet.
[INFO][abstract_intensifier.py:595] Added config b3f935 and rejected config ce0554 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config b99ec7 and rejected config b3f935 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 50 trials.
[INFO][smbo.py:320] Finished 100 trials.
[INFO][abstract_intensifier.py:595] Added config 5487e8 and rejected config b99ec7 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 92ab40 and rejected config 5487e8 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 17b438 and rejected config 92ab40 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 150 trials.
[INFO][abstract_intensifier.py:595] Added config 90142d and rejected config 17b438 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 55a07b and rejected config 90142d as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config b467f4 and rejected config 55a07b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 200 trials.
[INFO][abstract_intensifier.py:595] Added config 65117b and rejected config b467f4 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config aa3524 and rejected config 65117b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 1ca4da and rejected config aa3524 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 76f610 and rejected config 1ca4da as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:328] Configuration budget is exhausted:
[INFO][smbo.py:329] --- Remaining wallclock time: inf
[INFO][smbo.py:330] --- Remaining cpu time: inf
[INFO][smbo.py:331] --- Remaining trials: 0
[INFO][abstract_intensifier.py:306] Using only one seed for deterministic scenario.
Default cost: 50.0
Incumbent cost: 8.451187731769955e-09

from __future__ import annotations

import numpy as np
from ConfigSpace import Configuration, ConfigurationSpace, Float
from matplotlib import pyplot as plt

from smac import MultiFidelityFacade, RunHistory, Scenario
from smac.intensifier.hyperband_utils import get_n_trials_for_hyperband_multifidelity

__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"


class QuadraticFunction:
    max_budget = 500

    @property
    def configspace(self) -> ConfigurationSpace:
        cs = ConfigurationSpace(seed=0)
        x = Float("x", (-5, 5), default=-5)
        cs.add([x])

        return cs

    def train(self, config: Configuration, seed: int = 0, budget: float | None = None) -> float:
        """Returns the y value of a quadratic function with a minimum we know to be at x=0."""
        x = config["x"]

        if budget is None:
            multiplier = 1
        else:
            multiplier = 1 + budget / self.max_budget

        return x**2 * multiplier


def plot(runhistory: RunHistory, incumbent: Configuration) -> None:
    plt.figure()

    # Plot ground truth
    x = list(np.linspace(-5, 5, 100))
    y = [xi * xi for xi in x]
    plt.plot(x, y)

    # Plot all trials
    for k, v in runhistory.items():
        config = runhistory.get_config(k.config_id)
        x = config["x"]
        y = v.cost  # type: ignore
        plt.scatter(x, y, c="blue", alpha=0.1, zorder=9999, marker="o")

    # Plot incumbent
    plt.scatter(incumbent["x"], incumbent["x"] * incumbent["x"], c="red", zorder=10000, marker="x")

    plt.show()


if __name__ == "__main__":
    model = QuadraticFunction()

    min_budget = 10  # minimum budget per trial
    max_budget = 500  # maximum budget per trial
    eta = 3  # standard HB parameter influencing the number of stages

    # Let's calculate how many trials we need to exhaust the total optimization budget (in terms of
    # fidelity units)
    n_trials = get_n_trials_for_hyperband_multifidelity(
        total_budget=10000,  # this is the total optimization budget we specify in terms of fidelity units
        min_budget=min_budget,  # This influences the Hyperband rounds, minimum budget per trial
        max_budget=max_budget,  # This influences the Hyperband rounds, maximum budget per trial
        eta=eta,  # This influences the Hyperband rounds
        print_summary=True,
    )

    # Scenario object specifying the optimization "environment"
    scenario = Scenario(
        model.configspace, deterministic=True, n_trials=n_trials, min_budget=min_budget, max_budget=max_budget
    )

    # Now we use SMAC to find the best hyperparameters
    smac = MultiFidelityFacade(
        scenario,
        model.train,  # We pass the target function here
        overwrite=True,  # Overrides any previous results that are found that are inconsistent with the meta-data
        intensifier=MultiFidelityFacade.get_intensifier(scenario=scenario, eta=eta),
    )

    incumbent = smac.optimize()

    # Get cost of default configuration
    default_cost = smac.validate(model.configspace.get_default_configuration())
    print(f"Default cost: {default_cost}")

    # Let's calculate the cost of the incumbent
    incumbent_cost = smac.validate(incumbent)
    print(f"Incumbent cost: {incumbent_cost}")

    # Let's plot it too
    plot(smac.runhistory, incumbent)

previous

Stochastic Gradient Descent On Multiple Datasets

next

Multi-Objective