Advanced examples

Optimizing a small Convolutional Neural Network on MNIST

In this example we will cover the functionalities:

1) A more realistic Search Space
2) Live logging
3) Visualization

The first two parts are covered in example 5 while the visualization is presented in example 6 and example 7

1) A more realistic Search Space

So far, the search space in the quickstart examples consisted of a single continuous variables. The interesting problems have, of course, more difficult spaces to search through. Challenges include

  1. higher number of dimensions
  2. not only continuous parameters, but also integer values, categorical choices
  3. dependencies/conditionalities between parameters

The worker for the MNIST example has a fairly complex search space that covers the challenges above. Please checkout the Keras or the PyTorch to see how the search space and how it is encoded.

2) Live logging

So far, the results of a run were only available after it finished. For very long and expensive runs, one might want to check the progress while the optimization is still in progress. To that end, HpBandSter includes a functionality that we call live logging. The idea is simply to write enough information to disk such that intermediate results can be analysed the same way as completed runs. The storage is implemented by the result logger which stores two JSON files in a specified directory. Loading that information is implemented in this function

To store the information simply create a result logger and pass it to an optimizer. The relevant lines from the example are

import hpbandster.core.result as hpres
result_logger = hpres.json_result_logger(directory=args.shared_directory, overwrite=False)
bohb = BOHB(  configspace = worker.get_configspace(),
			  run_id = args.run_id,
			  host=host,
			  nameserver=ns_host,
			  nameserver_port=ns_port,
			  result_logger=result_logger,
			  min_budget=args.min_budget, max_budget=args.max_budget, 
		   )

Please check out example to see how to load these results.

3) Visualization

HpBandster contains a few tools to visualize the results of an optimization run. An example for the static plots can be found in example 6. The presented quantities allow one to see how many configurations were evaluated on which budgets, how many concurrent runs were done during the optimization, and how the losses correlated across budgets.

There is also a small interactive plot tool shown in example 7. It allows to better explore the performance of the runs by plotting various iterations and allowing to show more information on selected configurations. It is not the most elaborate tool, but maybe it can help to find patterns in the results.

Continuing a Run or Warmstarting the Optimization

Under certain circumstances, one might want to continue a run, or reuse old evaluations for a slightly modified objective. Those include, but are not limited to

  • The run ended unexpectedly due to some hardware problems (computer crashed).
  • The analysis reveals that the optimization was still improving meaning the run was to short.
  • The maximum budget used in the run turned out to be insufficient to achieve the desired performance.
  • The finished run might have been an preliminary run on only a subset of the data to speed up the experiment. The obtained results are still good proxies for the full dataset.

In example 8, we show how one loads the result of an old run and initialize the model for a new run. Specifically, we optimize the same CNN architecture on MNIST, but train on more data points. We also increased the minimum budget, as we do not expect anymore gains from evaluations after one epoch.

Note

While some specific details of the problem can change, it is important to realize that this only works if the search space is identical, and the old objective is still a very good approximation for the new one! Continuing with the same worker, settings, and budgets using this functionality will not recover runs that where pending when the JSON files where written. Also, the iterations start counting at zero again, i.e. it might use different budgets as the next iteration of the previous run would have used.