Specify Number of Trials via a Total Budget in Hyperband

This example uses a dummy function but illustrates how to setup Hyperband if you want to specify a total optimization budget in terms of fidelity units.

In Hyperband, normally SMAC calculates a typical Hyperband round. If the number of trials is not used up by one single round, the next round is started. Instead of specifying the number of trial beforehand, specify the total budget in terms of the fidelity units and let SMAC calculate how many trials that would be.

------------------------------ HYPERBAND IN MULTI-FIDELITY ------------------------------
total budget:            10000
total number of trials:  249
number of HB rounds:     3.802816901408451

        ~~~~~~~~~~~~HYPERBAND ROUND
        eta:                                     3
        min budget per trial:                    10
        max budget per trial:                    500
        total number of trials per HB round:     69
        budget used per HB round:                2629.6296296296296
        number of brackets:                      4
        budgets per stage:                       {0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0], 1: [55.55555555555555, 166.66666666666666, 500.0], 2: [166.66666666666666, 500.0], 3: [500.0]}
        n configs per stage:                     {0: [27, 9, 3, 1], 1: [12, 4, 1], 2: [6, 2], 3: [4]}
-----------------------------------------------------------------------------------------
[INFO][abstract_initial_design.py:147] Using 10 initial design configurations and 0 additional configurations.
[INFO][abstract_intensifier.py:306] Using only one seed for deterministic scenario.
[INFO][successive_halving.py:164] Successive Halving uses budget type BUDGETS with eta 3, min budget 10, and max budget 500.
[INFO][successive_halving.py:323] Number of configs in stage:
[INFO][successive_halving.py:325] --- Bracket 0: [27, 9, 3, 1]
[INFO][successive_halving.py:325] --- Bracket 1: [12, 4, 1]
[INFO][successive_halving.py:325] --- Bracket 2: [6, 2]
[INFO][successive_halving.py:325] --- Bracket 3: [4]
[INFO][successive_halving.py:327] Budgets in stage:
[INFO][successive_halving.py:329] --- Bracket 0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 1: [55.55555555555555, 166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 2: [166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 3: [500.0]
[INFO][abstract_intensifier.py:516] Added config 837184 as new incumbent because there are no incumbents yet.
[INFO][abstract_intensifier.py:595] Added config 7dea39 and rejected config 837184 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 568310 and rejected config 7dea39 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 50 trials.
[INFO][smbo.py:320] Finished 100 trials.
[INFO][abstract_intensifier.py:595] Added config a9f98d and rejected config 568310 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 8feb9a and rejected config a9f98d as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 2bef77 and rejected config 8feb9a as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 150 trials.
[INFO][abstract_intensifier.py:595] Added config 18e24c and rejected config 2bef77 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 1811f9 and rejected config 18e24c as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config a6306e and rejected config 1811f9 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 9363dc and rejected config a6306e as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 200 trials.
[INFO][abstract_intensifier.py:595] Added config d0eba5 and rejected config 9363dc as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config c64d86 and rejected config d0eba5 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:328] Configuration budget is exhausted:
[INFO][smbo.py:329] --- Remaining wallclock time: inf
[INFO][smbo.py:330] --- Remaining cpu time: inf
[INFO][smbo.py:331] --- Remaining trials: 0
[INFO][abstract_intensifier.py:306] Using only one seed for deterministic scenario.
Default cost: 50.0
Incumbent cost: 5.948347972367606e-08

from __future__ import annotations

import numpy as np
from ConfigSpace import Configuration, ConfigurationSpace, Float
from matplotlib import pyplot as plt

from smac import MultiFidelityFacade, RunHistory, Scenario
from smac.intensifier.hyperband_utils import get_n_trials_for_hyperband_multifidelity

__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"


class QuadraticFunction:
    max_budget = 500

    @property
    def configspace(self) -> ConfigurationSpace:
        cs = ConfigurationSpace(seed=0)
        x = Float("x", (-5, 5), default=-5)
        cs.add([x])

        return cs

    def train(self, config: Configuration, seed: int = 0, budget: float | None = None) -> float:
        """Returns the y value of a quadratic function with a minimum we know to be at x=0."""
        x = config["x"]

        if budget is None:
            multiplier = 1
        else:
            multiplier = 1 + budget / self.max_budget

        return x**2 * multiplier


def plot(runhistory: RunHistory, incumbent: Configuration) -> None:
    plt.figure()

    # Plot ground truth
    x = list(np.linspace(-5, 5, 100))
    y = [xi * xi for xi in x]
    plt.plot(x, y)

    # Plot all trials
    for k, v in runhistory.items():
        config = runhistory.get_config(k.config_id)
        x = config["x"]
        y = v.cost  # type: ignore
        plt.scatter(x, y, c="blue", alpha=0.1, zorder=9999, marker="o")

    # Plot incumbent
    plt.scatter(incumbent["x"], incumbent["x"] * incumbent["x"], c="red", zorder=10000, marker="x")

    plt.show()


if __name__ == "__main__":
    model = QuadraticFunction()

    min_budget = 10  # minimum budget per trial
    max_budget = 500  # maximum budget per trial
    eta = 3  # standard HB parameter influencing the number of stages

    # Let's calculate how many trials we need to exhaust the total optimization budget (in terms of
    # fidelity units)
    n_trials = get_n_trials_for_hyperband_multifidelity(
        total_budget=10000,  # this is the total optimization budget we specify in terms of fidelity units
        min_budget=min_budget,  # This influences the Hyperband rounds, minimum budget per trial
        max_budget=max_budget,  # This influences the Hyperband rounds, maximum budget per trial
        eta=eta,  # This influences the Hyperband rounds
        print_summary=True,
    )

    # Scenario object specifying the optimization "environment"
    scenario = Scenario(
        model.configspace, deterministic=True, n_trials=n_trials, min_budget=min_budget, max_budget=max_budget
    )

    # Now we use SMAC to find the best hyperparameters
    smac = MultiFidelityFacade(
        scenario,
        model.train,  # We pass the target function here
        overwrite=True,  # Overrides any previous results that are found that are inconsistent with the meta-data
        intensifier=MultiFidelityFacade.get_intensifier(scenario=scenario, eta=eta),
    )

    incumbent = smac.optimize()

    # Get cost of default configuration
    default_cost = smac.validate(model.configspace.get_default_configuration())
    print(f"Default cost: {default_cost}")

    # Let's calculate the cost of the incumbent
    incumbent_cost = smac.validate(incumbent)
    print(f"Incumbent cost: {incumbent_cost}")

    # Let's plot it too
    plot(smac.runhistory, incumbent)

previous

Stochastic Gradient Descent On Multiple Datasets

next

Multi-Objective