Note
Click here to download the full example code
Ask-and-Tell¶
This examples show how to use the Ask-and-Tell interface.
[INFO][abstract_initial_design.py:81] Reducing the number of initial configurations from 20 to 10 (max_ratio == 0.1).
[INFO][abstract_initial_design.py:133] Using 10 initial design and 0 additional configurations.
[INFO][intensifier.py:275] No incumbent provided in the first run. Sampling a new challenger...
[INFO][intensifier.py:446] First run and no incumbent provided. Challenger is assumed to be the incumbent.
[INFO][intensifier.py:566] Updated estimated cost of incumbent on 1 trials: 1102.7878
[INFO][abstract_intensifier.py:340] Challenger (657.2534) is better than incumbent (1102.7878) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -0.9968221839517355 -> -1.3192483183561738
[INFO][abstract_intensifier.py:367] --- x1: 4.30847043171525 -> 4.293599385932769
[INFO][abstract_intensifier.py:340] Challenger (508.0605) is better than incumbent (657.2534) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3192483183561738 -> -1.322176500365123
[INFO][abstract_intensifier.py:367] --- x1: 4.293599385932769 -> 3.990176539976968
[INFO][abstract_intensifier.py:340] Challenger (505.5016) is better than incumbent (508.0605) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.322176500365123 -> -1.3192483183561738
[INFO][abstract_intensifier.py:367] --- x1: 3.990176539976968 -> 3.976758315033301
[INFO][abstract_intensifier.py:340] Challenger (503.7298) is better than incumbent (505.5016) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3192483183561738 -> -1.3185110815210224
[INFO][abstract_intensifier.py:367] --- x1: 3.976758315033301 -> 3.9708564245469855
[INFO][abstract_intensifier.py:340] Challenger (500.0745) is better than incumbent (503.7298) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3185110815210224 -> -1.3223619688098793
[INFO][abstract_intensifier.py:367] --- x1: 3.9708564245469855 -> 3.972784094963705
[INFO][abstract_intensifier.py:340] Challenger (365.9729) is better than incumbent (500.0745) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3223619688098793 -> -1.3252199860300689
[INFO][abstract_intensifier.py:367] --- x1: 3.972784094963705 -> 3.655066271214169
[INFO][abstract_intensifier.py:340] Challenger (179.7681) is better than incumbent (365.9729) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x1: 3.655066271214169 -> 3.0766680957028107
[INFO][abstract_intensifier.py:340] Challenger (101.2001) is better than incumbent (179.7681) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3252199860300689 -> -1.2879784021263396
[INFO][abstract_intensifier.py:367] --- x1: 3.0766680957028107 -> 2.638507107817224
[INFO][abstract_intensifier.py:340] Challenger (73.7738) is better than incumbent (101.2001) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.2879784021263396 -> -1.2798289749424758
[INFO][abstract_intensifier.py:367] --- x1: 2.638507107817224 -> 2.466069498133966
[INFO][abstract_intensifier.py:340] Challenger (69.8772) is better than incumbent (73.7738) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.2798289749424758 -> -1.2799170414725793
[INFO][abstract_intensifier.py:367] --- x1: 2.466069498133966 -> 2.442420995658015
[INFO][abstract_intensifier.py:340] Challenger (65.381) is better than incumbent (69.8772) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.2799170414725793 -> -1.2906867715983639
[INFO][abstract_intensifier.py:367] --- x1: 2.442420995658015 -> 2.4413316791645077
[INFO][abstract_intensifier.py:340] Challenger (55.8803) is better than incumbent (65.381) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.2906867715983639 -> -1.3148355606771056
[INFO][abstract_intensifier.py:367] --- x1: 2.4413316791645077 -> 2.4395798258018546
[INFO][abstract_intensifier.py:340] Challenger (5.7265) is better than incumbent (55.8803) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3148355606771056 -> -1.312939445981982
[INFO][abstract_intensifier.py:367] --- x1: 2.4395798258018546 -> 1.7851965671304857
[INFO][abstract_intensifier.py:340] Challenger (5.4511) is better than incumbent (5.7265) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.312939445981982 -> -1.31339138148163
[INFO][abstract_intensifier.py:367] --- x1: 1.7851965671304857 -> 1.6934798613411255
[INFO][abstract_intensifier.py:340] Challenger (5.3782) is better than incumbent (5.4511) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.31339138148163 -> -1.3116263745498071
[INFO][abstract_intensifier.py:367] --- x1: 1.6934798613411255 -> 1.7017559227898733
[INFO][abstract_intensifier.py:340] Challenger (5.3609) is better than incumbent (5.3782) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3116263745498071 -> -1.3118488592394675
[INFO][abstract_intensifier.py:367] --- x1: 1.7017559227898733 -> 1.7082015256333234
[INFO][abstract_intensifier.py:340] Challenger (5.3482) is better than incumbent (5.3609) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3118488592394675 -> -1.3087626949250475
[INFO][abstract_intensifier.py:367] --- x1: 1.7082015256333234 -> 1.7262195165302554
[INFO][abstract_intensifier.py:340] Challenger (5.3482) is better than incumbent (5.3482) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3087626949250475 -> -1.3087967283792028
[INFO][abstract_intensifier.py:340] Challenger (5.3371) is better than incumbent (5.3482) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x0: -1.3087967283792028 -> -1.3087626949250475
[INFO][abstract_intensifier.py:367] --- x1: 1.7262195165302554 -> 1.7046672922919566
[INFO][base_smbo.py:260] Configuration budget is exhausted.
[INFO][abstract_facade.py:325] Final Incumbent: {'x0': -1.3087626949250475, 'x1': 1.7046672922919566}
[INFO][abstract_facade.py:326] Estimated cost: 5.3370968860138115
Default cost: 16916.0
Default cost: 5.3370968860138115
from ConfigSpace import Configuration, ConfigurationSpace, Float
from smac import HyperparameterOptimizationFacade, Scenario
from smac.runhistory.dataclasses import TrialValue
__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"
class Rosenbrock2D:
@property
def configspace(self) -> ConfigurationSpace:
cs = ConfigurationSpace(seed=0)
x0 = Float("x0", (-5, 10), default=-3)
x1 = Float("x1", (-5, 10), default=-4)
cs.add_hyperparameters([x0, x1])
return cs
def train(self, config: Configuration, seed: int = 0) -> float:
"""The 2-dimensional Rosenbrock function as a toy model.
The Rosenbrock function is well know in the optimization community and
often serves as a toy problem. It can be defined for arbitrary
dimensions. The minimium is always at x_i = 1 with a function value of
zero. All input parameters are continuous. The search domain for
all x's is the interval [-5, 10].
"""
x1 = config["x0"]
x2 = config["x1"]
cost = 100.0 * (x2 - x1**2.0) ** 2.0 + (1 - x1) ** 2.0
return cost
if __name__ == "__main__":
model = Rosenbrock2D()
# Scenario object
scenario = Scenario(model.configspace, deterministic=False, n_trials=100)
intensifier = HyperparameterOptimizationFacade.get_intensifier(
scenario,
max_config_calls=1, # We basically use one seed only
)
# Now we use SMAC to find the best hyperparameters
smac = HyperparameterOptimizationFacade(
scenario,
model.train,
intensifier=intensifier,
overwrite=True,
)
# We can ask SMAC which trials should be evaluated next
for _ in range(10):
info = smac.ask()
assert info.seed is not None
cost = model.train(info.config, seed=info.seed)
value = TrialValue(cost=cost, time=0.5)
smac.tell(info, value)
# After calling ask+tell, we can still optimize
incumbent = smac.optimize()
assert smac.stats.finished == 100
# Get cost of default configuration
default_cost = smac.validate(model.configspace.get_default_configuration())
print(f"Default cost: {default_cost}")
# Let's calculate the cost of the incumbent
incumbent_cost = smac.validate(incumbent)
print(f"Default cost: {incumbent_cost}")
Total running time of the script: ( 0 minutes 24.008 seconds)