benchmark

class PD1Config
dataclass

Bases: Config

The config for PD1.

class PD1Result
dataclass

Bases: Result[PD1Config, int]

score: float
prop

The score of interest.

error: float
prop

The error of interest.

val_score: float
prop

The score on the validation set.

val_error: float
prop

The error on the validation set.

cost: float
prop

The train cost of the model (asssumed to be seconds).

Please double check with YAHPO.

class PD1ResultSimple
dataclass

Bases: PD1Result

Used for all PD1 benchmarks, except imagenet, lm1b, translate_wmt, uniref50.

test_score: float
prop

The score on the test set.

test_error: float
prop

The error on the test set.

class PD1ResultTransformer
dataclass

Bases: PD1Result

Imagenet, lm1b, translate_wmt, uniref50, cifar100 contains no test error.

test_score: float
prop

The score on the test set.

test_error: float
prop

The error on the test set.

class PD1Benchmark(*, datadir=None, seed=None, prior=None, perturb_prior=None)

Bases: Benchmark[C, R, int]

PARAMETER	DESCRIPTION
datadir	Path to the data directory TYPE: str | Path | None DEFAULT: None
seed	The seed to use for the space TYPE: int | None DEFAULT: None
prior	Any prior to use for the benchmark TYPE: str | Path | C | Mapping[str, Any] | None DEFAULT: None
perturb_prior	Whether to perturb the prior. If specified, this is interpreted as the std of a normal from which to perturb numerical hyperparameters of the prior, and the raw probability of swapping a categorical value. TYPE: float | None DEFAULT: None

Source code in src/mfpbench/pd1/benchmark.py

def __init__(
    self,
    *,
    datadir: str | Path | None = None,
    seed: int | None = None,
    prior: str | Path | C | Mapping[str, Any] | None = None,
    perturb_prior: float | None = None,
):
    """Create a PD1 Benchmark.

    Args:
        datadir: Path to the data directory
        seed: The seed to use for the space
        prior: Any prior to use for the benchmark
        perturb_prior: Whether to perturb the prior. If specified, this
            is interpreted as the std of a normal from which to perturb
            numerical hyperparameters of the prior, and the raw probability
            of swapping a categorical value.
    """
    cls = self.__class__
    space = cls._create_space(seed=seed)
    name = f"{cls.pd1_dataset}-{cls.pd1_model}-{cls.pd1_batchsize}"
    if datadir is None:
        datadir = PD1Source.default_location()

    datadir = Path(datadir) if isinstance(datadir, str) else datadir
    if not datadir.exists():
        raise FileNotFoundError(
            f"Can't find folder at {datadir}."
            f"\n`python -m mfpbench download --status --data-dir {datadir.parent}`",
        )
    self._surrogates: dict[str, XGBRegressor] | None = None
    self.datadir = datadir

    super().__init__(
        seed=seed,
        name=name,
        prior=prior,
        perturb_prior=perturb_prior,
        space=space,
    )

pd1_dataset: str
classvar

The dataset that this benchmark uses.

pd1_model: str
classvar

The model that this benchmark uses.

pd1_batchsize: int
classvar

The batchsize that this benchmark uses.

pd1_metrics: tuple[str, ...]
classvar

The metrics that are available for this benchmark.

Config: type[C]
attr

The config type for this benchmark.

Result: type[R]
attr

The result type for this benchmark.

surrogates: dict[str, XGBRegressor]
prop

The surrogates for this benchmark, one per metric.

surrogate_dir: Path
prop

The directory where the surrogates are stored.

surrogate_paths: dict[str, Path]
prop

The paths to the surrogates.

def load()

Load the benchmark.

Source code in src/mfpbench/pd1/benchmark.py

def load(self) -> None:
    """Load the benchmark."""
    _ = self.surrogates  # Call up the surrogate into memory