2D Schaffer Function with Objective Weights

A simple example on how to use multi-objective optimization is shown. The schaffer function is used.

[INFO][abstract_initial_design.py:133] Using 10 initial design and 0 additional configurations.
[INFO][intensifier.py:275] No incumbent provided in the first run. Sampling a new challenger...
[INFO][intensifier.py:446] First run and no incumbent provided. Challenger is assumed to be the incumbent.
[INFO][intensifier.py:566] Updated estimated cost of incumbent on 1 trials: 1.0
[INFO][abstract_intensifier.py:340] Challenger (0.3333) is better than incumbent (0.5888) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -0.9324859157204628 -> 1.3451112993061543
[INFO][abstract_intensifier.py:340] Challenger (0.2725) is better than incumbent (0.2725) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3451112993061543 -> 1.3448614852608838
[INFO][abstract_intensifier.py:340] Challenger (0.2724) is better than incumbent (0.2725) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3448614852608838 -> 1.3442250183165956
[INFO][abstract_intensifier.py:340] Challenger (0.2724) is better than incumbent (0.2724) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3442250183165956 -> 1.344081913330525
[INFO][abstract_intensifier.py:340] Challenger (0.2724) is better than incumbent (0.2724) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.344081913330525 -> 1.3439457083376847
[INFO][abstract_intensifier.py:340] Challenger (0.2723) is better than incumbent (0.2724) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3439457083376847 -> 1.3436009486299065
[INFO][abstract_intensifier.py:340] Challenger (0.2723) is better than incumbent (0.2723) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3436009486299065 -> 1.343062445203584
[INFO][abstract_intensifier.py:340] Challenger (0.2723) is better than incumbent (0.2723) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.343062445203584 -> 1.3430038657074754
[INFO][abstract_intensifier.py:340] Challenger (0.2722) is better than incumbent (0.2723) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3430038657074754 -> 1.3423393633152223
[INFO][abstract_intensifier.py:340] Challenger (0.2722) is better than incumbent (0.2722) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3423393633152223 -> 1.3423268024390436
[INFO][abstract_intensifier.py:340] Challenger (0.2721) is better than incumbent (0.2722) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3423268024390436 -> 1.3411748493907347
[INFO][abstract_intensifier.py:340] Challenger (0.2721) is better than incumbent (0.2721) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3411748493907347 -> 1.3408257098322904
[INFO][abstract_intensifier.py:340] Challenger (0.264) is better than incumbent (0.2721) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: 1.3408257098322904 -> -1.154695986884652
[INFO][abstract_intensifier.py:340] Challenger (0.264) is better than incumbent (0.264) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.154695986884652 -> -1.1581912672166585
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.264) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1581912672166585 -> -1.1828007982514523
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1828007982514523 -> -1.1749832250073124
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1749832250073124 -> -1.1748448120914996
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1748448120914996 -> -1.1727127330103513
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1727127330103513 -> -1.1727241873269154
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1727241873269154 -> -1.1730755740919152
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1730755740919152 -> -1.1731571705331119
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1731571705331119 -> -1.173357486075275
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.173357486075275 -> -1.173430891047234
[INFO][abstract_intensifier.py:340] Challenger (0.2639) is better than incumbent (0.2639) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.173430891047234 -> -1.1736659846779394
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2599) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1736659846779394 -> -1.199505468821445
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.199505468821445 -> -1.1868767052586038
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1868767052586038 -> -1.1881219618652126
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1881219618652126 -> -1.1886192958480413
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1886192958480413 -> -1.188960306897171
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.188960306897171 -> -1.1891790281629118
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1891790281629118 -> -1.1897144592927509
[INFO][abstract_intensifier.py:340] Challenger (0.2598) is better than incumbent (0.2598) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.1897144592927509 -> -1.193068449279342
[INFO][abstract_intensifier.py:340] Challenger (0.2389) is better than incumbent (0.2406) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.193068449279342 -> -1.24515924007905
[INFO][abstract_intensifier.py:340] Challenger (0.2388) is better than incumbent (0.2389) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.24515924007905 -> -1.2504825004110844
[INFO][abstract_intensifier.py:340] Challenger (0.2387) is better than incumbent (0.2388) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2504825004110844 -> -1.2617113515399638
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2387) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2617113515399638 -> -1.2868030299177242
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2868030299177242 -> -1.284268280183599
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.284268280183599 -> -1.2832225932589822
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2832225932589822 -> -1.2797239850921116
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2797239850921116 -> -1.2795249484497022
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2795249484497022 -> -1.2796848158374088
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2796848158374088 -> -1.2796788948519637
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2796788948519637 -> -1.2796139308001808
[INFO][abstract_intensifier.py:340] Challenger (0.2386) is better than incumbent (0.2386) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2796139308001808 -> -1.279623543617439
[INFO][abstract_intensifier.py:340] Challenger (0.2327) is better than incumbent (0.2328) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.279623543617439 -> -1.311138660898445
[INFO][abstract_intensifier.py:340] Challenger (0.2327) is better than incumbent (0.2327) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.311138660898445 -> -1.2972804523461572
[INFO][abstract_intensifier.py:340] Challenger (0.2327) is better than incumbent (0.2327) on 1 trials.
[INFO][abstract_intensifier.py:364] Changes in incumbent:
[INFO][abstract_intensifier.py:367] --- x: -1.2972804523461572 -> -1.2980629453226524
[INFO][base_smbo.py:260] Configuration budget is exhausted.
[INFO][abstract_facade.py:325] Final Incumbent: {'x': -1.2980629453226524}
[INFO][abstract_facade.py:326] Estimated cost: 0.23265567918059796
Default costs: [0. 4.]
Incumbent costs: [1.68496741 0.49271563]

from __future__ import annotations

from typing import Dict, Tuple

import numpy as np
from ConfigSpace import Configuration, ConfigurationSpace
from matplotlib import pyplot as plt

from smac import HyperparameterOptimizationFacade as HPOFacade
from smac import Scenario
from smac.facade import AbstractFacade

__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
__license__ = "3-clause BSD"


def schaffer(x: float) -> Tuple[float, float]:
    f1 = np.square(x)
    f2 = np.square(np.sqrt(f1) - 2)

    return f1, f2


def target_function(config: Configuration, seed: int = 0) -> Dict[str, float]:
    f1, f2 = schaffer(config["x"])
    return {"metric1": f1, "metric2": f2}


def plot(all_x: list[float]) -> None:
    plt.figure()
    for x in all_x:
        f1, f2 = schaffer(x)
        plt.scatter(f1, f2, c="blue", alpha=0.1, zorder=3000)

    # plt.vlines([1], 0, 4, linestyles="dashed", colors=["red"])
    # plt.hlines([1], 0, 4, linestyles="dashed", colors=["red"])

    plt.show()


def plot_from_smac(smac: AbstractFacade) -> None:
    rh = smac.runhistory
    all_x = []
    for trial_key in rh:
        config = rh.ids_config[trial_key.config_id]
        all_x.append(config["x"])

    plot(all_x)


if __name__ == "__main__":
    # Simple configspace
    cs = ConfigurationSpace({"x": (-2.0, 2.0)})

    # Scenario object
    scenario = Scenario(
        configspace=cs,
        deterministic=True,  # Only one seed
        n_trials=150,
        objectives=["metric1", "metric2"],
    )

    smac = HPOFacade(
        scenario=scenario,
        target_function=target_function,
        multi_objective_algorithm=HPOFacade.get_multi_objective_algorithm(
            scenario,
            objective_weights=[1, 2],  # Weight metric2 twice as much as metric1
        ),
        overwrite=True,
    )
    incumbent = smac.optimize()

    default_cost = smac.validate(cs.get_default_configuration())
    print(f"Default costs: {default_cost}")

    incumbent_cost = smac.validate(incumbent)
    print(f"Incumbent costs: {incumbent_cost}")

    # Plot the evaluated points
    plot_from_smac(smac)

previous

Multi-Objective

next

ParEGO