Api

API for the neps package.

run

run(
    evaluate_pipeline: (
        Callable[..., EvaluatePipelineReturn] | str
    ),
    pipeline_space: (
        Mapping[str, dict | str | int | float | Parameter]
        | SearchSpace
        | ConfigurationSpace
    ),
    *,
    root_directory: str | Path = "neps_results",
    overwrite_working_directory: bool = False,
    post_run_summary: bool = True,
    max_evaluations_total: int | None = None,
    max_evaluations_per_run: int | None = None,
    continue_until_max_evaluation_completed: bool = False,
    max_cost_total: int | float | None = None,
    ignore_errors: bool = False,
    objective_value_on_error: float | None = None,
    cost_value_on_error: float | None = None,
    sample_batch_size: int | None = None,
    optimizer: (
        OptimizerChoice
        | Mapping[str, Any]
        | tuple[OptimizerChoice, Mapping[str, Any]]
        | Callable[
            Concatenate[SearchSpace, ...], AskFunction
        ]
        | CustomOptimizer
        | Literal["auto"]
    ) = "auto"
) -> None

Run the optimization.

Parallelization

To run with multiple processes or machines, execute the script that calls neps.run() multiple times. They will keep in sync using the file-sytem, requiring that root_directory be shared between them.

import neps
import logging

logging.basicConfig(level=logging.INFO)

def evaluate_pipeline(some_parameter: float) -> float:
    validation_error = -some_parameter
    return validation_error

pipeline_space = dict(some_parameter=neps.Float(lower=0, upper=1))
neps.run(
    evaluate_pipeline=evaluate_pipeline,
    pipeline_space={
        "some_parameter": (0.0, 1.0),   # float
        "another_parameter": (0, 10),   # integer
        "optimizer": ["sgd", "adam"],   # categorical
        "epoch": neps.Integer(          # fidelity integer
            lower=1,
            upper=100,
            is_fidelity=True
        ),
        "learning_rate": neps.Float(    # log spaced float
            lower=1e-5,
            uperr=1,
            log=True
        ),
        "alpha": neps.Float(            # float with a prior
            lower=0.1,
            upper=1.0,
            prior=0.99,
            prior_confidence="high",
        )
    },
    root_directory="usage_example",
    max_evaluations_total=5,
)

PARAMETER	DESCRIPTION
evaluate_pipeline	The objective function to minimize. This will be called with a configuration from the pipeline_space= that you define. The function should return one of the following: A float, which is the objective value to minimize. A dict which can have the following keys: { "objective_to_minimize": float, # The thing to minimize (required) "cost": float, # The cost of the evaluate_pipeline, used by some algorithms "info_dict": dict, # Any additional information you want to store, should be YAML serializable } str usage for dynamic imports If a string, it should be in the format "/path/to/:function". to specify the function to call. You may also directly provide an mode to import, e.g., "my.module.something:evaluate_pipeline". TYPE: Callable[..., EvaluatePipelineReturn] | str
pipeline_space	The search space to minimize over. This most direct way to specify the search space is as follows: neps.run( pipeline_space={ "dataset": "mnist", # constant "nlayers": (2, 10), # integer "alpha": (0.1, 1.0), # float "optimizer": [ # categorical "adam", "sgd", "rmsprop" ], "learning_rate": neps.Float(, # log spaced float lower=1e-5, upper=1, log=True ), "epochs": neps.Integer( # fidelity integer lower=1, upper=100, is_fidelity=True ), "batch_size": neps.Integer( # integer with a prior lower=32, upper=512, prior=128 ), } ) You can also directly instantiate any of the parameters defined by Parameter and provide them directly. Some important properties you can set on parameters are: prior=: If you have a good idea about what a good setting for a parameter may be, you can set this as the prior for a parameter. You can specify this along with prior_confidence if you would like to assign a "low", "medium", or "high" confidence to the prior. Yaml support To support spaces defined in yaml, you may also define the parameters as dictionarys, e.g., neps.run( pipeline_space={ "dataset": "mnist", "nlayers": {"type": "int", "lower": 2, "upper": 10}, "alpha": {"type": "float", "lower": 0.1, "upper": 1.0}, "optimizer": {"type": "cat", "choices": ["adam", "sgd", "rmsprop"]}, "learning_rate": {"type": "float", "lower": 1e-5, "upper": 1, "log": True}, "epochs": {"type": "int", "lower": 1, "upper": 100, "is_fidelity": True}, "batch_size": {"type": "int", "lower": 32, "upper": 512, "prior": 128}, } ) ConfigSpace support You may also use a ConfigurationSpace object from the ConfigSpace library. TYPE: Mapping[str, dict | str | int | float | Parameter] | SearchSpace | ConfigurationSpace
root_directory	The directory to save progress to. TYPE: str | Path DEFAULT: 'neps_results'
overwrite_working_directory	If true, delete the working directory at the start of the run. This is, e.g., useful when debugging a evaluate_pipeline function. TYPE: bool DEFAULT: False
post_run_summary	If True, creates a csv file after each worker is done, holding summary information about the configs and results. TYPE: bool DEFAULT: True
max_evaluations_per_run	Number of evaluations this specific call should do. TYPE: int | None DEFAULT: None
max_evaluations_total	Number of evaluations after which to terminate. This is shared between all workers operating in the same root_directory. TYPE: int | None DEFAULT: None
continue_until_max_evaluation_completed	If true, only stop after max_evaluations_total have been completed. This is only relevant in the parallel setting. TYPE: bool DEFAULT: False
max_cost_total	No new evaluations will start when this cost is exceeded. Requires returning a cost in the evaluate_pipeline function, e.g., return dict(loss=loss, cost=cost). TYPE: int | float | None DEFAULT: None
ignore_errors	Ignore hyperparameter settings that threw an error and do not raise an error. Error configs still count towards max_evaluations_total. TYPE: bool DEFAULT: False
objective_value_on_error	Setting this and cost_value_on_error to any float will supress any error and will use given objective_to_minimize value instead. default: None TYPE: float | None DEFAULT: None
cost_value_on_error	Setting this and objective_value_on_error to any float will supress any error and will use given cost value instead. default: None TYPE: float | None DEFAULT: None
sample_batch_size	The number of samples to ask for in a single call to the optimizer. When to use this? This is only useful in scenarios where you have many workers available, and the optimizers sample time prevents full worker utilization, as can happen with Bayesian optimizers. In this case, the currently active worker will first check if there are any new configurations to evaluate, and if not, generate sample_batch_size new configurations that the proceeding workers will then pick up and evaluate. We advise to only use this if: You are using a "ifbo" or "bayesian_optimization". You have a fast to evaluate evaluate_pipeline You have a significant amount of workers available, relative to the time it takes to evaluate a single configuration. Downsides of batching The primary downside of batched optimization is that the next sample_batch_size configurations will not be able to take into account the results of any new evaluations, even if they were to come in relatively quickly. TYPE: int | None DEFAULT: None
optimizer	Which optimizer to use. Not sure which to use? Leave this at "auto" and neps will choose the optimizer based on the search space given. Available optimizers "bayesian_optimization", Models the relation between hyperparameters in your pipeline_space and the results of evaluate_pipeline using bayesian optimization. This acts as a cheap surrogate model of you evaluate_pipeline function that can be used for optimization. When to use this? Bayesion optimization is a good general purpose choice, especially if the size of your search space is not too large. It is also the best option to use if you do not have or want to use a fidelity parameter. Note that acquiring the next configuration to evaluate with bayesian optimization can become prohibitvely expensive as the number of configurations evaluated increases. If there is some numeric cost associated with evaluating a configuration, you can provide this as a cost when returning the results from your evaluate_pipeline function. By specifying cost_aware=True, the optimizer will attempt to balance getting the best result while minimizing the cost. If you have priors, we recommend looking at pibo. PARAMETER DESCRIPTION space The search space to sample from. TYPE: SearchSpace initial_design_size Number of samples used before using the surrogate model. If "ndim", it will use the number of parameters in the search space. TYPE: int | Literal['ndim'] DEFAULT: 'ndim' cost_aware Whether to consider reported "cost" from configurations in decision making. If True, the optimizer will weigh potential candidates by how much they cost, incentivising the optimizer to explore cheap, good performing configurations. This amount is modified over time. If "log", the cost will be log-transformed before being used. Warning If using cost, cost must be provided in the reports of the trials. TYPE: bool | Literal['log'] DEFAULT: False device Device to use for the optimization. TYPE: device | str | None DEFAULT: None "ifbo" A transformer that has been trained to predict loss curves of deep-learing models, used to guide the optimization procedure and select configurations which are most promising to evaluate. When to use this? Use this when you think that early signal in your loss curve could be used to distinguish which configurations are likely to achieve a good performance. This algorithm will take many small steps in evaluating your configuration so we also advise that saving and loading your model checkpoint should be relatively fast. This algorithm requires a fidelity parameter, such as epochs, to be present. Each time we evaluate a configuration, we will only evaluate it for a single epoch, before returning back to the ifbo algorithm to select the next configuration. Fidelities? A fidelity parameter lets you control how many resources to invest in a single evaluation. For example, a common one for deep-learing is epochs. We can evaluate a model for just a single epoch, (fidelity step) to gain more information about the model's performance and decide what to do next. Paper: openreview.net/forum?id=VyoY3Wh9Wd Github: github.com/automl/ifBO/tree/main PARAMETER DESCRIPTION pipeline_space Space in which to search TYPE: SearchSpace step_size The size of the step to take in the fidelity domain. TYPE: int | float DEFAULT: 1 sample_prior_first Whether to sample the default configuration first TYPE: bool DEFAULT: False initial_design_size Number of configs to sample before starting optimization If None, the number of configs will be equal to the number of dimensions. TYPE: int | Literal['ndim'] DEFAULT: 'ndim' device Device to use for the model TYPE: device | str | None DEFAULT: None surrogate_path Path to the surrogate model to use TYPE: str | Path | None DEFAULT: None surrogate_version Version of the surrogate model to use TYPE: str DEFAULT: '0.0.1' "successive_halving": A bandit-based optimization algorithm that uses a fidelity parameter to gradually invest resources into more promising configurations. Fidelities? A fidelity parameter lets you control how many resources to invest in a single evaluation. For example, a common one for deep-learing is epochs. By evaluating a model for just a few epochs, we can quickly get a sense if the model is promising or not. Only those that perform well get promoted and evaluated at a higher epoch. When to use this? When you think that the rank of N configurations at a lower fidelity correlates very well with the rank if you were to evaluate those configurations at higher fidelities. It does this by creating a competition between N configurations and racing them in a bracket against each other. This bracket has a series of incrementing rungs, where lower rungs indicate less resources invested. The amount of resources is related to your fidelity parameter, with the highest rung relating to the maximum of your fidelity parameter. Those that perform well get promoted and evaluated with more resources. # A bracket indicating the rungs and configurations. # Those which performed best get promoted through the rungs. | | fidelity | c1 | c2 | c3 | c4 | c5 | ... | cN | | Rung 0 | (3 epochs) | o | o | o | o | o | ... | o | | Rung 1 | (9 epochs) | o | | o | o | | ... | o | | Rung 2 | (27 epochs) | o | | | | | ... | | By default, new configurations are sampled using a uniform distribution, however you can also specify to prefer sampling from around a distribution you think is more promising by setting the prior and the prior_confidence of parameters of your search space. You can choose between these by setting sampler="uniform" or sampler="prior". PARAMETER DESCRIPTION space The search space to sample from. TYPE: SearchSpace eta The reduction factor used for building brackets TYPE: int DEFAULT: 3 early_stopping_rate Determines the number of rungs in a bracket Choosing 0 creates maximal rungs given the fidelity bounds. TYPE: int DEFAULT: 0 sampler The type of sampling procedure to use: If "uniform", samples uniformly from the space when it needs to sample. If "prior", samples from the prior distribution built from the prior and prior_confidence values in the search space. TYPE: Literal['uniform', 'prior'] DEFAULT: 'uniform' sample_prior_first Whether to sample the prior configuration first, and if so, should it be at the highest fidelity level. TYPE: bool | Literal['highest_fidelity'] DEFAULT: False "hyperband": Another bandit-based optimization algorithm that uses a fidelity parameter, very similar to successive_halving, but hedges a bit more on the safe side, just incase your fidelity parameters isn't as well correlated as you'd like. When to use this? Use this when you think lower fidelity evaluations of your configurations carries some signal about their ranking at higher fidelities, but not enough to be certain Hyperband is like Successive Halving but it instead of always having the same bracket layout, it runs different brackets with different rungs. This helps hedge against scenarios where rankings at the lowest fidelity do not correlate well with the upper fidelity. # Hyperband runs different successive halving brackets | Bracket 1 | | Bracket 2 | | Bracket 3 | | Rung 0 | ... | | (skipped) | | (skipped) | | Rung 1 | ... | | Rung 1 | ... | | (skipped) | | Rung 2 | ... | | Rung 2 | ... | | Rung 2 | ... | For more information, see the successive_halving documentation, as this algorithm could be considered an extension of it. PARAMETER DESCRIPTION space The search space to sample from. TYPE: SearchSpace eta The reduction factor used for building brackets TYPE: int DEFAULT: 3 sampler The type of sampling procedure to use: If "uniform", samples uniformly from the space when it needs to sample. If "prior", samples from the prior distribution built from the prior and prior_confidence values in the search space. TYPE: Literal['uniform', 'prior'] DEFAULT: 'uniform' sample_prior_first Whether to sample the prior configuration first, and if so, should it be at the highest fidelity level. TYPE: bool | Literal['highest_fidelity'] DEFAULT: False "priorband": Priorband is also a bandit-based optimization algorithm that uses a fidelity, providing a general purpose sampling extension to other algorithms. It makes better use of the prior information you provide in the search space along with the fact that you can afford to explore and take more risk at lower fidelities. When to use this? Use this when you have a good idea of what good parameters look like and can specify them through the prior and prior_confidence parameters in the search space. As priorband is flexible, you may choose between the existing tradeoffs the other algorithms provide through the use of base=. Priorband works by adjusting the sampling procedure to sample from one of the following three distributions: 1) a uniform distribution 2) a prior distribution 3) a distribution around the best found configuration so far. By weighing the likelihood of good configurations having been sampled from each of these distribution, we can score them against each other to aid selection. We further use the fact that we can afford to explore and take more risk at lower fidelities, which is factored into the sampling procedure. See: openreview.net/forum?id=uoiwugtpCH&noteId=xECpK2WH6k PARAMETER DESCRIPTION space The search space to sample from. TYPE: SearchSpace eta The reduction factor used for building brackets TYPE: int DEFAULT: 3 sample_prior_first Whether to sample the prior configuration first. TYPE: bool | Literal['highest_fidelity'] DEFAULT: False base The base algorithm to use for the bracketing. TYPE: Literal['successive_halving', 'hyperband', 'asha', 'async_hb'] DEFAULT: 'hyperband' bayesian_optimization_kick_in_point If a number N, after N * maximum_fidelity worth of fidelity has been evaluated, proceed with bayesian optimization when sampling a new configuration. TYPE: int | float | None DEFAULT: None "asha": A bandit-based optimization algorithm that uses a fidelity parameter, the asynchronous version of successive_halving. one that scales better to many parallel workers. When to use this? Use this when you think lower fidelity evaluations of your configurations carries a strong signal about their ranking at higher fidelities, and you have many workers available to evaluate configurations in parallel. It does this by maintaining one big bracket, i.e. one big on-going competition, with a promotion rule based on the sizes of each rung. # ASHA maintains one big bracket with an exponentially decreasing amount of # configurations promoted, relative to those in the rung below. | | fidelity | c1 | c2 | c3 | c4 | c5 | ... | Rung 0 | (3 epochs) | o | o | o | o | o | ... | Rung 1 | (9 epochs) | o | | o | o | | ... | Rung 2 | (27 epochs) | o | | | o | | ... For more information, see the successive_halving documentation, as this algorithm could be considered an extension of it. PARAMETER DESCRIPTION space The search space to sample from. TYPE: SearchSpace eta The reduction factor used for building brackets TYPE: int DEFAULT: 3 sampler The type of sampling procedure to use: If "uniform", samples uniformly from the space when it needs to sample. If "prior", samples from the prior distribution built from the prior and prior_confidence values in the search space. TYPE: Literal['uniform', 'prior'] DEFAULT: 'uniform' sample_prior_first Whether to sample the prior configuration first, and if so, should it be at the highest fidelity. TYPE: bool | Literal['highest_fidelity'] DEFAULT: False "async_hb": An asynchronous version of hyperband, where the brackets are run asynchronously, and the promotion rule is based on the number of evaluations each configuration has had. When to use this? Use this when you think lower fidelity evaluations of your configurations carries some signal about their ranking at higher fidelities, but not confidently, and you have many workers available to evaluate configurations in parallel. # Async HB runs different "asha" brackets, which are unbounded in the number # of configurations that can be in each. The bracket chosen at each iteration # is a sampling function based on the resources invested in each bracket. | Bracket 1 | | Bracket 2 | | Bracket 3 | | Rung 0 | ... | (skipped) | | (skipped) | | Rung 1 | ... | Rung 1 | ... | (skipped) | | Rung 2 | ... | Rung 2 | ... | Rung 2 | ... For more information, see the hyperband documentation, successive_halving documentation, and the asha documentation, as this algorithm takes elements from each. PARAMETER DESCRIPTION space The search space to sample from. TYPE: SearchSpace eta The reduction factor used for building brackets TYPE: int DEFAULT: 3 sampler The type of sampling procedure to use: If "uniform", samples uniformly from the space when it needs to sample. If "prior", samples from the prior distribution built from the prior and prior_confidence values in the search space. TYPE: Literal['uniform', 'prior'] DEFAULT: 'uniform' sample_prior_first Whether to sample the prior configuration first. TYPE: bool DEFAULT: False "random_search": A simple random search algorithm that samples configurations uniformly at random. You may also use_priors= to sample from a distribution centered around your defined priors. PARAMETER DESCRIPTION pipeline_space The search space to sample from. TYPE: SearchSpace use_priors Whether to use priors when sampling. TYPE: bool DEFAULT: False ignore_fidelity Whether to ignore fidelity when sampling. In this case, the max fidelity is always used. TYPE: bool DEFAULT: True "grid_search": A simple grid search algorithm which discretizes the search space and evaluates all possible configurations. PARAMETER DESCRIPTION pipeline_space The search space to sample from. TYPE: SearchSpace With any optimizer choice, you also may provide some additional parameters to the optimizers. We do not recommend this unless you are familiar with the optimizer you are using. You may also specify an optimizer as a dictionary for supporting reading in serialized yaml formats: neps.run( ..., optimzier={ "name": "priorband", "sample_prior_first": True, } ) Own optimzier Lastly, you may also provide your own optimizer which must satisfy the AskFunction signature. class MyOpt: def __init__(self, space: SearchSpace): ... def __call__( self, trials: Mapping[str, Trial], budget_info: BudgetInfo | None, n: int | None = None, ) -> SampledConfig | list[SampledConfig]: # Sample a new configuration. # # Args: # trials: All of the trials that are known about. # budget_info: information about the budget constraints. # # Returns: # The sampled configuration(s) neps.run( ..., optimizer=MyOpt, ) This is mainly meant for internal development but allows you to use the NePS runtime to run your optimizer. TYPE: OptimizerChoice | Mapping[str, Any] | tuple[OptimizerChoice, Mapping[str, Any]] | Callable[Concatenate[SearchSpace, ...], AskFunction] | CustomOptimizer | Literal['auto'] DEFAULT: 'auto'

Source code in neps\api.py

def run(  # noqa: PLR0913
    evaluate_pipeline: Callable[..., EvaluatePipelineReturn] | str,
    pipeline_space: (
        Mapping[str, dict | str | int | float | Parameter]
        | SearchSpace
        | ConfigurationSpace
    ),
    *,
    root_directory: str | Path = "neps_results",
    overwrite_working_directory: bool = False,
    post_run_summary: bool = True,
    max_evaluations_total: int | None = None,
    max_evaluations_per_run: int | None = None,
    continue_until_max_evaluation_completed: bool = False,
    max_cost_total: int | float | None = None,
    ignore_errors: bool = False,
    objective_value_on_error: float | None = None,
    cost_value_on_error: float | None = None,
    sample_batch_size: int | None = None,
    optimizer: (
        OptimizerChoice
        | Mapping[str, Any]
        | tuple[OptimizerChoice, Mapping[str, Any]]
        | Callable[Concatenate[SearchSpace, ...], AskFunction]
        | CustomOptimizer
        | Literal["auto"]
    ) = "auto",
) -> None:
    """Run the optimization.

    !!! tip "Parallelization"

        To run with multiple processes or machines, execute the script that
        calls `neps.run()` multiple times. They will keep in sync using
        the file-sytem, requiring that `root_directory` be shared between them.


    ```python
    import neps
    import logging

    logging.basicConfig(level=logging.INFO)

    def evaluate_pipeline(some_parameter: float) -> float:
        validation_error = -some_parameter
        return validation_error

    pipeline_space = dict(some_parameter=neps.Float(lower=0, upper=1))
    neps.run(
        evaluate_pipeline=evaluate_pipeline,
        pipeline_space={
            "some_parameter": (0.0, 1.0),   # float
            "another_parameter": (0, 10),   # integer
            "optimizer": ["sgd", "adam"],   # categorical
            "epoch": neps.Integer(          # fidelity integer
                lower=1,
                upper=100,
                is_fidelity=True
            ),
            "learning_rate": neps.Float(    # log spaced float
                lower=1e-5,
                uperr=1,
                log=True
            ),
            "alpha": neps.Float(            # float with a prior
                lower=0.1,
                upper=1.0,
                prior=0.99,
                prior_confidence="high",
            )
        },
        root_directory="usage_example",
        max_evaluations_total=5,
    )
    ```

    Args:
        evaluate_pipeline: The objective function to minimize. This will be called
            with a configuration from the `pipeline_space=` that you define.

            The function should return one of the following:

            * A `float`, which is the objective value to minimize.
            * A `dict` which can have the following keys:

                ```python
                {
                    "objective_to_minimize": float,  # The thing to minimize (required)
                    "cost": float,  # The cost of the evaluate_pipeline, used by some algorithms
                    "info_dict": dict,  # Any additional information you want to store, should be YAML serializable
                }
                ```

            ??? note "`str` usage for dynamic imports"

                If a string, it should be in the format `"/path/to/:function"`.
                to specify the function to call. You may also directly provide
                an mode to import, e.g., `"my.module.something:evaluate_pipeline"`.

        pipeline_space: The search space to minimize over.

            This most direct way to specify the search space is as follows:

            ```python
            neps.run(
                pipeline_space={
                    "dataset": "mnist",             # constant
                    "nlayers": (2, 10),             # integer
                    "alpha": (0.1, 1.0),            # float
                    "optimizer": [                  # categorical
                        "adam", "sgd", "rmsprop"
                    ],
                    "learning_rate": neps.Float(,   # log spaced float
                        lower=1e-5, upper=1, log=True
                    ),
                    "epochs": neps.Integer(         # fidelity integer
                        lower=1, upper=100, is_fidelity=True
                    ),
                    "batch_size": neps.Integer(     # integer with a prior
                        lower=32, upper=512, prior=128
                    ),

                }
            )
            ```

            You can also directly instantiate any of the parameters
            defined by [`Parameter`][neps.space.parameters.Parameter]
            and provide them directly.

            Some important properties you can set on parameters are:

            * `prior=`: If you have a good idea about what a good setting
                for a parameter may be, you can set this as the prior for
                a parameter. You can specify this along with `prior_confidence`
                if you would like to assign a `"low"`, `"medium"`, or `"high"`
                confidence to the prior.


            !!! note "Yaml support"

                To support spaces defined in yaml, you may also define the parameters
                as dictionarys, e.g.,

                ```python
                neps.run(
                    pipeline_space={
                        "dataset": "mnist",
                        "nlayers": {"type": "int", "lower": 2, "upper": 10},
                        "alpha": {"type": "float", "lower": 0.1, "upper": 1.0},
                        "optimizer": {"type": "cat", "choices": ["adam", "sgd", "rmsprop"]},
                        "learning_rate": {"type": "float", "lower": 1e-5, "upper": 1, "log": True},
                        "epochs": {"type": "int", "lower": 1, "upper": 100, "is_fidelity": True},
                        "batch_size": {"type": "int", "lower": 32, "upper": 512, "prior": 128},
                    }
                )
                ```

            !!! note "ConfigSpace support"

                You may also use a `ConfigurationSpace` object from the
                `ConfigSpace` library.

        root_directory: The directory to save progress to.

        overwrite_working_directory: If true, delete the working directory at the start of
            the run. This is, e.g., useful when debugging a evaluate_pipeline function.

        post_run_summary: If True, creates a csv file after each worker is done,
            holding summary information about the configs and results.

        max_evaluations_per_run: Number of evaluations this specific call should do.

        max_evaluations_total: Number of evaluations after which to terminate.
            This is shared between all workers operating in the same `root_directory`.

        continue_until_max_evaluation_completed:
            If true, only stop after max_evaluations_total have been completed.
            This is only relevant in the parallel setting.

        max_cost_total: No new evaluations will start when this cost is exceeded. Requires
            returning a cost in the evaluate_pipeline function, e.g.,
            `return dict(loss=loss, cost=cost)`.
        ignore_errors: Ignore hyperparameter settings that threw an error and do not raise
            an error. Error configs still count towards max_evaluations_total.
        objective_value_on_error: Setting this and cost_value_on_error to any float will
            supress any error and will use given objective_to_minimize value instead. default: None
        cost_value_on_error: Setting this and objective_value_on_error to any float will
            supress any error and will use given cost value instead. default: None

        sample_batch_size:
            The number of samples to ask for in a single call to the optimizer.

            ??? tip "When to use this?"

                This is only useful in scenarios where you have many workers
                available, and the optimizers sample time prevents full
                worker utilization, as can happen with Bayesian optimizers.

                In this case, the currently active worker will first
                check if there are any new configurations to evaluate,
                and if not, generate `sample_batch_size` new configurations
                that the proceeding workers will then pick up and evaluate.

                We advise to only use this if:

                * You are using a `#!python "ifbo"` or `#!python "bayesian_optimization"`.
                * You have a fast to evaluate `evaluate_pipeline`
                * You have a significant amount of workers available, relative to the
                time it takes to evaluate a single configuration.

            ??? warning "Downsides of batching"

                The primary downside of batched optimization is that
                the next `sample_batch_size` configurations will not
                be able to take into account the results of any new
                evaluations, even if they were to come in relatively
                quickly.

        optimizer: Which optimizer to use.

            Not sure which to use? Leave this at `"auto"` and neps will
            choose the optimizer based on the search space given.

            ??? note "Available optimizers"

                ---

                * `#!python "bayesian_optimization"`,

                    ::: neps.optimizers.algorithms.bayesian_optimization
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "ifbo"`

                    ::: neps.optimizers.algorithms.ifbo
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "successive_halving"`:

                    ::: neps.optimizers.algorithms.successive_halving
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "hyperband"`:

                    ::: neps.optimizers.algorithms.hyperband
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "priorband"`:

                    ::: neps.optimizers.algorithms.priorband
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "asha"`:

                    ::: neps.optimizers.algorithms.asha
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "async_hb"`:

                    ::: neps.optimizers.algorithms.async_hb
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "random_search"`:

                    ::: neps.optimizers.algorithms.random_search
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---

                * `#!python "grid_search"`:

                    ::: neps.optimizers.algorithms.grid_search
                        options:
                            show_root_heading: false
                            show_signature: false
                            show_source: false

                ---


            With any optimizer choice, you also may provide some additional parameters to the optimizers.
            We do not recommend this unless you are familiar with the optimizer you are using. You
            may also specify an optimizer as a dictionary for supporting reading in serialized yaml
            formats:

            ```python
            neps.run(
                ...,
                optimzier={
                    "name": "priorband",
                    "sample_prior_first": True,
                }
            )
            ```

            ??? tip "Own optimzier"

                Lastly, you may also provide your own optimizer which must satisfy
                the [`AskFunction`][neps.optimizers.optimizer.AskFunction] signature.

                ```python
                class MyOpt:

                    def __init__(self, space: SearchSpace):
                        ...

                    def __call__(
                        self,
                        trials: Mapping[str, Trial],
                        budget_info: BudgetInfo | None,
                        n: int | None = None,
                    ) -> SampledConfig | list[SampledConfig]:
                        # Sample a new configuration.
                        #
                        # Args:
                        #   trials: All of the trials that are known about.
                        #   budget_info: information about the budget constraints.
                        #
                        # Returns:
                        #   The sampled configuration(s)


                neps.run(
                    ...,
                    optimizer=MyOpt,
                )
                ```

                This is mainly meant for internal development but allows you to use the NePS
                runtime to run your optimizer.

    """  # noqa: E501
    if (
        max_evaluations_total is None
        and max_evaluations_per_run is None
        and max_cost_total is None
    ):
        warnings.warn(
            "None of the following were set, this will run idefinitely until the worker"
            " process is stopped."
            f"\n * {max_evaluations_total=}"
            f"\n * {max_evaluations_per_run=}"
            f"\n * {max_cost_total=}",
            UserWarning,
            stacklevel=2,
        )

    logger.info(f"Starting neps.run using root directory {root_directory}")
    space = convert_to_space(pipeline_space)
    _optimizer_ask, _optimizer_info = load_optimizer(optimizer=optimizer, space=space)

    _eval: Callable
    if isinstance(evaluate_pipeline, str):
        module, funcname = evaluate_pipeline.rsplit(":", 1)
        eval_pipeline = dynamic_load_object(module, funcname)
        if not callable(eval_pipeline):
            raise ValueError(
                f"'{funcname}' in module '{module}' is not a callable function."
            )
        _eval = eval_pipeline
    elif callable(evaluate_pipeline):
        _eval = evaluate_pipeline
    else:
        raise ValueError(
            "evaluate_pipeline must be a callable or a string in the format"
            "'module:function'."
        )

    _launch_runtime(
        evaluation_fn=_eval,  # type: ignore
        optimizer=_optimizer_ask,
        optimizer_info=_optimizer_info,
        max_cost_total=max_cost_total,
        optimization_dir=Path(root_directory),
        max_evaluations_total=max_evaluations_total,
        max_evaluations_for_worker=max_evaluations_per_run,
        continue_until_max_evaluation_completed=continue_until_max_evaluation_completed,
        objective_value_on_error=objective_value_on_error,
        cost_value_on_error=cost_value_on_error,
        ignore_errors=ignore_errors,
        overwrite_optimization_dir=overwrite_working_directory,
        sample_batch_size=sample_batch_size,
    )

    if post_run_summary:
        full_frame_path, short_path = post_run_csv(root_directory)
        logger.info(
            "The post run summary has been created, which is a csv file with the "
            "output of all data in the run."
            f"\nYou can find a full dataframe at: {full_frame_path}."
            f"\nYou can find a quick summary at: {short_path}."
        )
    else:
        logger.info(
            "Skipping the creation of the post run summary, which is a csv file with the "
            " output of all data in the run."
            "\nSet `post_run_summary=True` to enable it."
        )