Note
Go to the end to download the full example code. or to run this example in your browser via Binder
Specify Number of Trials via a Total Budget in Hyperband¶
This example uses a dummy function but illustrates how to setup Hyperband if you want to specify a total optimization budget in terms of fidelity units.
In Hyperband, normally SMAC calculates a typical Hyperband round. If the number of trials is not used up by one single round, the next round is started. Instead of specifying the number of trial beforehand, specify the total budget in terms of the fidelity units and let SMAC calculate how many trials that would be.
![3 specify HB via total budget](../../_images/sphx_glr_3_specify_HB_via_total_budget_001.png)
------------------------------ HYPERBAND IN MULTI-FIDELITY ------------------------------
total budget: 10000
total number of trials: 249
number of HB rounds: 3.802816901408451
~~~~~~~~~~~~HYPERBAND ROUND
eta: 3
min budget per trial: 10
max budget per trial: 500
total number of trials per HB round: 69
budget used per HB round: 2629.6296296296296
number of brackets: 3
budgets per stage: {0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0], 1: [55.55555555555555, 166.66666666666666, 500.0], 2: [166.66666666666666, 500.0], 3: [500.0]}
n configs per stage: {0: [27, 9, 3, 1], 1: [12, 4, 1], 2: [6, 2], 3: [4]}
-----------------------------------------------------------------------------------------
[INFO][abstract_initial_design.py:147] Using 10 initial design configurations and 0 additional configurations.
[INFO][abstract_intensifier.py:306] Using only one seed for deterministic scenario.
[INFO][successive_halving.py:164] Successive Halving uses budget type BUDGETS with eta 3, min budget 10, and max budget 500.
[INFO][successive_halving.py:323] Number of configs in stage:
[INFO][successive_halving.py:325] --- Bracket 0: [27, 9, 3, 1]
[INFO][successive_halving.py:325] --- Bracket 1: [12, 4, 1]
[INFO][successive_halving.py:325] --- Bracket 2: [6, 2]
[INFO][successive_halving.py:325] --- Bracket 3: [4]
[INFO][successive_halving.py:327] Budgets in stage:
[INFO][successive_halving.py:329] --- Bracket 0: [18.51851851851852, 55.55555555555555, 166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 1: [55.55555555555555, 166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 2: [166.66666666666666, 500.0]
[INFO][successive_halving.py:329] --- Bracket 3: [500.0]
[INFO][abstract_intensifier.py:516] Added config ce0554 as new incumbent because there are no incumbents yet.
[INFO][abstract_intensifier.py:595] Added config b3f935 and rejected config ce0554 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config b99ec7 and rejected config b3f935 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 50 trials.
[INFO][smbo.py:320] Finished 100 trials.
[INFO][abstract_intensifier.py:595] Added config 5487e8 and rejected config b99ec7 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 92ab40 and rejected config 5487e8 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 17b438 and rejected config 92ab40 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 150 trials.
[INFO][abstract_intensifier.py:595] Added config 90142d and rejected config 17b438 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 55a07b and rejected config 90142d as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config b467f4 and rejected config 55a07b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:320] Finished 200 trials.
[INFO][abstract_intensifier.py:595] Added config 65117b and rejected config b467f4 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config aa3524 and rejected config 65117b as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 1ca4da and rejected config aa3524 as incumbent because it is not better than the incumbents on 1 instances:
[INFO][abstract_intensifier.py:595] Added config 76f610 and rejected config 1ca4da as incumbent because it is not better than the incumbents on 1 instances:
[INFO][smbo.py:328] Configuration budget is exhausted:
[INFO][smbo.py:329] --- Remaining wallclock time: inf
[INFO][smbo.py:330] --- Remaining cpu time: inf
[INFO][smbo.py:331] --- Remaining trials: 0
[INFO][abstract_intensifier.py:306] Using only one seed for deterministic scenario.
Default cost: 50.0
Incumbent cost: 8.451187731769955e-09
from __future__ import annotations
import numpy as np
from ConfigSpace import Configuration, ConfigurationSpace, Float
from matplotlib import pyplot as plt
from smac import MultiFidelityFacade, RunHistory, Scenario
from smac.intensifier.hyperband_utils import get_n_trials_for_hyperband_multifidelity
__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"
class QuadraticFunction:
max_budget = 500
@property
def configspace(self) -> ConfigurationSpace:
cs = ConfigurationSpace(seed=0)
x = Float("x", (-5, 5), default=-5)
cs.add([x])
return cs
def train(self, config: Configuration, seed: int = 0, budget: float | None = None) -> float:
"""Returns the y value of a quadratic function with a minimum we know to be at x=0."""
x = config["x"]
if budget is None:
multiplier = 1
else:
multiplier = 1 + budget / self.max_budget
return x**2 * multiplier
def plot(runhistory: RunHistory, incumbent: Configuration) -> None:
plt.figure()
# Plot ground truth
x = list(np.linspace(-5, 5, 100))
y = [xi * xi for xi in x]
plt.plot(x, y)
# Plot all trials
for k, v in runhistory.items():
config = runhistory.get_config(k.config_id)
x = config["x"]
y = v.cost # type: ignore
plt.scatter(x, y, c="blue", alpha=0.1, zorder=9999, marker="o")
# Plot incumbent
plt.scatter(incumbent["x"], incumbent["x"] * incumbent["x"], c="red", zorder=10000, marker="x")
plt.show()
if __name__ == "__main__":
model = QuadraticFunction()
min_budget = 10 # minimum budget per trial
max_budget = 500 # maximum budget per trial
eta = 3 # standard HB parameter influencing the number of stages
# Let's calculate how many trials we need to exhaust the total optimization budget (in terms of
# fidelity units)
n_trials = get_n_trials_for_hyperband_multifidelity(
total_budget=10000, # this is the total optimization budget we specify in terms of fidelity units
min_budget=min_budget, # This influences the Hyperband rounds, minimum budget per trial
max_budget=max_budget, # This influences the Hyperband rounds, maximum budget per trial
eta=eta, # This influences the Hyperband rounds
print_summary=True,
)
# Scenario object specifying the optimization "environment"
scenario = Scenario(
model.configspace, deterministic=True, n_trials=n_trials, min_budget=min_budget, max_budget=max_budget
)
# Now we use SMAC to find the best hyperparameters
smac = MultiFidelityFacade(
scenario,
model.train, # We pass the target function here
overwrite=True, # Overrides any previous results that are found that are inconsistent with the meta-data
intensifier=MultiFidelityFacade.get_intensifier(scenario=scenario, eta=eta),
)
incumbent = smac.optimize()
# Get cost of default configuration
default_cost = smac.validate(model.configspace.get_default_configuration())
print(f"Default cost: {default_cost}")
# Let's calculate the cost of the incumbent
incumbent_cost = smac.validate(incumbent)
print(f"Incumbent cost: {incumbent_cost}")
# Let's plot it too
plot(smac.runhistory, incumbent)
Total running time of the script: (0 minutes 8.283 seconds)