Note
Click here to download the full example code
Ask-and-Tell¶
This examples show how to use the Ask-and-Tell interface.
[INFO][abstract_initial_design.py:81] Reducing the number of initial configurations from 20 to 10 (max_ratio == 0.1).
[INFO][abstract_initial_design.py:133] Using 10 initial design and 0 additional configurations.
[INFO][intensifier.py:275] No incumbent provided in the first run. Sampling a new challenger...
[INFO][intensifier.py:446] First run and no incumbent provided. Challenger is assumed to be the incumbent.
[INFO][intensifier.py:566] Updated estimated cost of incumbent on 1 trials: 1102.7878
[INFO][abstract_intensifier.py:340] Challenger (700.6438) is better than incumbent (1102.7878) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.9968221839517355 -> -0.0801787059894652
[INFO][abstract_intensifier.py:367] --- x1: 4.30847043171525 -> 2.6511913893559456
[INFO][abstract_intensifier.py:340] Challenger (423.3081) is better than incumbent (700.6438) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.0801787059894652 -> -0.5748483514276161
[INFO][abstract_intensifier.py:367] --- x1: 2.6511913893559456 -> 2.3818598505133917
[INFO][abstract_intensifier.py:340] Challenger (392.447) is better than incumbent (423.3081) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.5748483514276161 -> -0.5571954690033651
[INFO][abstract_intensifier.py:367] --- x1: 2.3818598505133917 -> 2.285364622945317
[INFO][abstract_intensifier.py:340] Challenger (373.356) is better than incumbent (392.447) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.5571954690033651 -> -0.5136112514398299
[INFO][abstract_intensifier.py:367] --- x1: 2.285364622945317 -> 2.1901013339486397
[INFO][abstract_intensifier.py:340] Challenger (341.9607) is better than incumbent (373.356) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.5136112514398299 -> -0.5216817868049262
[INFO][abstract_intensifier.py:367] --- x1: 2.1901013339486397 -> 2.115098301281617
[INFO][abstract_intensifier.py:340] Challenger (314.5626) is better than incumbent (341.9607) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.5216817868049262 -> -0.520999279200856
[INFO][abstract_intensifier.py:367] --- x1: 2.115098301281617 -> 2.038497452566906
[INFO][abstract_intensifier.py:340] Challenger (311.7474) is better than incumbent (314.5626) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.520999279200856 -> -0.5235838293620301
[INFO][abstract_intensifier.py:367] --- x1: 2.038497452566906 -> 2.033191299635524
[INFO][abstract_intensifier.py:340] Challenger (311.1308) is better than incumbent (311.7474) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.5235838293620301 -> -0.5317202403312171
[INFO][abstract_intensifier.py:367] --- x1: 2.033191299635524 -> 2.039953327639248
[INFO][abstract_intensifier.py:340] Challenger (208.9282) is better than incumbent (311.1308) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.5317202403312171 -> -0.578728392807137
[INFO][abstract_intensifier.py:367] --- x1: 2.039953327639248 -> 1.7717139425912283
[INFO][abstract_intensifier.py:340] Challenger (41.7306) is better than incumbent (208.9282) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.578728392807137 -> 3.0288560991981086
[INFO][abstract_intensifier.py:367] --- x1: 1.7717139425912283 -> 8.56066371210313
[INFO][abstract_intensifier.py:340] Challenger (16.105) is better than incumbent (41.7306) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: 3.0288560991981086 -> 2.9836089266698727
[INFO][abstract_intensifier.py:367] --- x1: 8.56066371210313 -> 8.553062049166332
[INFO][abstract_intensifier.py:340] Challenger (7.2895) is better than incumbent (16.105) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: 2.9836089266698727 -> 2.9546639310050757
[INFO][abstract_intensifier.py:367] --- x1: 8.553062049166332 -> 8.543793205723064
[INFO][abstract_intensifier.py:340] Challenger (6.5272) is better than incumbent (7.2895) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: 2.9546639310050757 -> 2.9518405504114362
[INFO][abstract_intensifier.py:367] --- x1: 8.543793205723064 -> 8.548513010062864
[INFO][abstract_intensifier.py:340] Challenger (6.4147) is better than incumbent (6.5272) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: 2.9518405504114362 -> 2.9512680685155193
[INFO][abstract_intensifier.py:340] Challenger (4.0512) is better than incumbent (6.4147) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: 2.9512680685155193 -> 2.9318551326203552
[INFO][abstract_intensifier.py:367] --- x1: 8.548513010062864 -> 8.539278257778507
[INFO][abstract_intensifier.py:340] Challenger (3.7212) is better than incumbent (4.0512) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: 2.9318551326203552 -> 2.9239921707775345
[INFO][abstract_intensifier.py:367] --- x1: 8.539278257778507 -> 8.53577528269198
[INFO][abstract_intensifier.py:340] Challenger (3.7208) is better than incumbent (3.7212) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x1: 8.53577528269198 -> 8.563521674936085
[INFO][abstract_intensifier.py:340] Challenger (3.7127) is better than incumbent (3.7208) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x1: 8.563521674936085 -> 8.560208937943326
[INFO][abstract_intensifier.py:340] Challenger (3.7125) is better than incumbent (3.7127) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x1: 8.560208937943326 -> 8.560077337041246
[INFO][base_smbo.py:260] Configuration budget is exhausted.
[INFO][abstract_facade.py:325] Final Incumbent: {'x0': 2.9239921707775345, 'x1': 8.560077337041246}
[INFO][abstract_facade.py:326] Estimated cost: 3.71245216714634
Default cost: 16916.0
Default cost: 3.71245216714634
from ConfigSpace import Configuration, ConfigurationSpace, Float
from smac import HyperparameterOptimizationFacade, Scenario
from smac.runhistory.dataclasses import TrialValue
__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"
class Rosenbrock2D:
@property
def configspace(self) -> ConfigurationSpace:
cs = ConfigurationSpace(seed=0)
x0 = Float("x0", (-5, 10), default=-3)
x1 = Float("x1", (-5, 10), default=-4)
cs.add_hyperparameters([x0, x1])
return cs
def train(self, config: Configuration, seed: int = 0) -> float:
"""The 2-dimensional Rosenbrock function as a toy model.
The Rosenbrock function is well know in the optimization community and
often serves as a toy problem. It can be defined for arbitrary
dimensions. The minimium is always at x_i = 1 with a function value of
zero. All input parameters are continuous. The search domain for
all x's is the interval [-5, 10].
"""
x1 = config["x0"]
x2 = config["x1"]
cost = 100.0 * (x2 - x1**2.0) ** 2.0 + (1 - x1) ** 2.0
return cost
if __name__ == "__main__":
model = Rosenbrock2D()
# Scenario object
scenario = Scenario(model.configspace, deterministic=False, n_trials=100)
intensifier = HyperparameterOptimizationFacade.get_intensifier(
scenario,
max_config_calls=1, # We basically use one seed only
)
# Now we use SMAC to find the best hyperparameters
smac = HyperparameterOptimizationFacade(
scenario,
model.train,
intensifier=intensifier,
overwrite=True,
)
# We can ask SMAC which trials should be evaluated next
for _ in range(10):
info = smac.ask()
assert info.seed is not None
cost = model.train(info.config, seed=info.seed)
value = TrialValue(cost=cost, time=0.5)
smac.tell(info, value)
# After calling ask+tell, we can still optimize
incumbent = smac.optimize()
assert smac.stats.finished == 100
# Get cost of default configuration
default_cost = smac.validate(model.configspace.get_default_configuration())
print(f"Default cost: {default_cost}")
# Let's calculate the cost of the incumbent
incumbent_cost = smac.validate(incumbent)
print(f"Default cost: {incumbent_cost}")
Total running time of the script: ( 0 minutes 21.287 seconds)