Skip to content

Optimizers

Optimizers#

An Optimizer's goal is to achieve the optimal value for a given Metric or Metrics using repeated Trials.

What differentiates AMLTK from other optimization libraries is that we rely solely on optimizers that support an "Ask-and-Tell" interface. This means we can "Ask" and optimizer for its next suggested Trial, and we can "Tell" it a Report when we have one. In fact, here's the required interface.

class Optimizer:

    def tell(self, report: Trial.Report) -> None: ...

    def ask(self) -> Trial: ...

Now we do require optimizers to implement these ask() and tell() methods, correctly filling in a Trial with appropriate parsing out results from the Report, as this will be different for every optimizer.

Why only Ask and Tell Optimizers?

  1. Easy Parallelization: Many optimizers handle running the function to optimize and hence roll out their own parallelization schemes and store data in all various different ways. By taking this repsonsiblity away from an optimzer and giving it to the user, we can easily parallelize how we wish

  2. API maintenance: Many optimziers are research code and hence a bit unstable with resepct to their API so wrapping around them can be difficult. By requiring this "Ask-and-Tell" interface, we reduce the complexity of what is required of both the "Optimizer" and wrapping it.

  3. Full Integration: We can fully hook into the life cycle of a running optimizer. We are not relying on the optimizer to support callbacks at every step of their hot-loop and as such, we can fully leverage all the other systems of AutoML-toolkit

  4. Easy Integration: it makes developing and integrating new optimizers easy. You only have to worry that the internal state of the optimizer is updated accordingly to these two "Ask" and "Tell" events and that's it.

For a reference on implementing an optimizer you can refer to any of the following API Docs: * SMAC * NePs * Optuna * Random Search

Integrating your own#

The base Optimizer class, defines the API we require optimizers to implement.

  • ask() - Ask the optimizer for a new Trial to evaluate.
  • tell() - Tell the optimizer the result of the sampled config. This comes in the form of a Trial.Report.

Additionally, to aid users from switching between optimizers, the preferred_parser() method should return either a parser function or a string that can be used with node.search_space(parser=..._) to extract the search space for the optimizer.

Please refer to the code of Random Search on github for an example of how to implement a new optimizer.