The Function Approximation Toy Benchmark

Task: approximate a function at timestep t in each of one or multiple dimensions

Cost: distance between prediction and function

Number of hyperparameters to control: user specified starting at one integer with no fixed upper limit

State Information: Remaining budget, instance descriptions for each dimension (inflection point and slope), last action for each dimension

Noise Level: None

Instance space: one simple function (constant, sigmoid, logarithmic, quadratic, polynomial) per dimension. Sampling notebook and example datasets in repository.

This benchmark is not built on top of an algorithm, but is simply a function approximation task. In each step until the cutoff, the DAC controller predicts one y-value for a given function curve per task dimension. The predictions are can be configured to be discrete, in which case there is some distance between the true function value and the best possible prediction. This distance is used as a cost function. If multiple task dimensions are used, the total cost is computed by multiplying the costs of all dimensions. Furthermore, it is possible to assign different importance weights to each dimension, mirroring hyperparameter importances.

The benchmark is very cheap to run and the instances can be sampled and shaped easily. Therefore it’s a good starting point for any new DAC method or to gain specific insights for which fine control over the instance distribution is required.

The Sigmoid benchmark was constructed by Biedenkapp et al. for the paper `”Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework” <https://www.tnt.uni-hannover.de/papers/data/1432/20-ECAI-DAC.pdf>`_ at ECAI 2020 and later extended to include multiple function classes and importance weights.

Function Approximation Benchmark.

class dacbench.benchmarks.function_approximation_benchmark.FunctionApproximationBenchmark(config_path=None, config=None)

Bases: AbstractBenchmark

Benchmark with default configuration & relevant functions for Function Approximation.

get_benchmark(dimension=None, seed=0)

Get Sigmoid Benchmark from DAC paper.

Parameters:

• dimension (int) – Sigmoid dimension, was 1, 2, 3 or 5 in the paper

• seed (int) – Environment seed

• Returns

• -------

• env (SigmoidEnv) – Sigmoid environment

get_environment()

Return Function Approximation env with current configuration.

Returns:

FunctionApproximationEnv

Function Approximation environment

read_instance_set(test=False)

Read instance set from file.

Function Approximation Environment.

class dacbench.envs.function_approximation.FunctionApproximationEnv(config)

Bases: AbstractMADACEnv

Function Approximation Environment.

close()

Close environment.

get_default_reward(_)

Get default reward: muliply dimensions.

get_default_state(_)

Get default state representation.

get_sum_reward(_)

Get sum reward.

reset(seed=None, options=None)

Reset environment.

step(action)

Step function: check how close the action is to the target.

class dacbench.envs.function_approximation.FunctionApproximationInstance(functions: list[AbstractFunction], dimension_importances: list[float], discrete: list[bool], omit_instance_type: bool = False)

Bases: object

Function Approximation Instance.

previous

Benchmark Overview

next

The Luby Toy Benchmark