benchmark

The hartmann benchmarks.

The presets of terrible, bad, moderate and good are empirically obtained hyperparameters for the hartmann function

The function flattens with increasing fidelity bias. Along with increasing noise, that obviously makes one config harder to distinguish from another. Moreover, this works with any number of fidelitiy levels.

class MFHartmann3Config
dataclass

Bases: Config

def validate()

Validate this config.

Source code in src/mfpbench/synthetic/hartmann/benchmark.py

def validate(self) -> None:
    """Validate this config."""
    assert 0.0 <= self.X_0 <= 1.0
    assert 0.0 <= self.X_1 <= 1.0
    assert 0.0 <= self.X_2 <= 1.0

class MFHartmann6Config
dataclass

Bases: Config

def validate()

Validate this config.

Source code in src/mfpbench/synthetic/hartmann/benchmark.py

def validate(self) -> None:
    """Validate this config."""
    assert 0.0 <= self.X_0 <= 1.0
    assert 0.0 <= self.X_1 <= 1.0
    assert 0.0 <= self.X_2 <= 1.0
    assert 0.0 <= self.X_3 <= 1.0
    assert 0.0 <= self.X_4 <= 1.0
    assert 0.0 <= self.X_5 <= 1.0

class MFHartmannResult
dataclass

Bases: Result[C, int]

score: float
prop

The score of interest.

error: float
prop

The score of interest.

test_score: float
prop

Just returns the score.

test_error: float
prop

Just returns the error.

val_score: float
prop

Just returns the score.

val_error: float
prop

Just returns the error.

cost: float
prop

Just retuns the fidelity.

class MFHartmannBenchmark(*, seed=None, bias=None, noise=None, prior=None, perturb_prior=None)

Bases: Benchmark, Generic[G, C]

PARAMETER	DESCRIPTION
seed	The seed to use. TYPE: int | None DEFAULT: None
bias	How much bias to introduce TYPE: float | None DEFAULT: None
noise	How much noise to introduce TYPE: float | None DEFAULT: None
prior	The prior to use for the benchmark. if Path - path to a file if Mapping - Use directly if None - There is no prior TYPE: str | Path | C | Mapping[str, Any] | None DEFAULT: None
perturb_prior	If not None, will perturb the prior by this amount. For numericals, while for categoricals, this is interpreted as the probability of swapping the value for a random one. TYPE: float | None DEFAULT: None

Source code in src/mfpbench/synthetic/hartmann/benchmark.py

def __init__(
    self,
    *,
    seed: int | None = None,
    bias: float | None = None,
    noise: float | None = None,
    prior: str | Path | C | Mapping[str, Any] | None = None,
    perturb_prior: float | None = None,
):
    """Initialize the benchmark.

    Args:
        seed: The seed to use.
        bias: How much bias to introduce
        noise: How much noise to introduce
        prior: The prior to use for the benchmark.

            * if `Path` - path to a file
            * if `Mapping` - Use directly
            * if `None` - There is no prior

        perturb_prior: If not None, will perturb the prior by this amount.
            For numericals, while for categoricals, this is interpreted as
            the probability of swapping the value for a random one.
    """
    cls = self.__class__
    self.bias = bias if bias is not None else cls.mfh_bias_noise[0]
    self.noise = noise if noise is not None else cls.mfh_bias_noise[1]
    self.mfh = cls.Generator(
        n_fidelities=cls.fidelity_range[1],
        fidelity_noise=self.noise,
        fidelity_bias=self.bias,
        seed=seed,
    )

    name = (
        f"mfh{cls.mfh_dims}_{cls.mfh_suffix}"
        if cls.mfh_suffix != ""
        else f"mfh{cls.mfh_dims}"
    )
    space = ConfigurationSpace(name=name, seed=seed)
    space.add_hyperparameters(
        [
            UniformFloatHyperparameter(f"X_{i}", lower=0.0, upper=1.0)
            for i in range(cls.mfh_dims)
        ],
    )
    super().__init__(
        name=name,
        space=space,
        seed=seed,
        prior=prior,
        perturb_prior=perturb_prior,
    )

mfh_dims: int
classvar

How many dimensions there are to the Hartmann function.

mfh_suffix: str
classvar

Suffix for the benchmark name

Config: type[C]
attr

The Config type for this mfhartmann benchmark.

Generator: type[G]
attr

The underlying mfhartmann function generator.

mfh_bias_noise: tuple[float, float]
classvar

The default bias and noise for mfhartmann benchmarks.

optimum: C
prop

The optimum of the benchmark.