smac.runner.dask_runner¶
Classes¶
|
Interface to submit and collect a job in a distributed fashion. |
Interfaces¶
- class smac.runner.dask_runner.DaskParallelRunner(single_worker, patience=5, dask_client=None)[source]¶
Bases:
AbstractRunner
Interface to submit and collect a job in a distributed fashion. DaskParallelRunner is intended to comply with the bridge design pattern. Nevertheless, to reduce the amount of code within single-vs-parallel implementations, DaskParallelRunner wraps a BaseRunner object which is then executed in parallel on n_workers.
This class then is constructed by passing an AbstractRunner that implements a run method, and is capable of doing so in a serial fashion. Next, this wrapper class uses dask to initialize N number of AbstractRunner that actively wait of a TrialInfo to produce a RunInfo object.
To be more precise, the work model is then:
The intensifier dictates “what” to run (a configuration/instance/seed) via a TrialInfo object.
An abstract runner takes this TrialInfo object and launches the task via submit_run. In the case of DaskParallelRunner, n_workers receive a pickle-object of DaskParallelRunner.single_worker, each with a run method coming from DaskParallelRunner.single_worker.run()
TrialInfo objects are run in a distributed fashion, and their results are available locally to each worker. The result is collected by iter_results and then passed to SMBO.
Exceptions are also locally available to each worker and need to be collected.
Dask works with Future object which are managed via the DaskParallelRunner.client.
- Parameters:
single_worker (AbstractRunner) – A runner to run in a distributed fashion. Will be distributed using n_workers.
patience (int, default to 5) – How much to wait for workers (seconds) to be available if one fails.
dask_client (Client | None, defaults to None) – User-created dask client, which can be used to start a dask cluster and then attach SMAC to it. This will not be closed automatically and will have to be closed manually if provided explicitly. If none is provided (default), a local one will be created for you and closed upon completion.
- __del__()[source]¶
Makes sure that when this object gets deleted, the client is terminated. This is only done if the client was created by the dask runner.
- Return type:
None
- count_available_workers()[source]¶
Total number of workers available. This number is dynamic as more resources can be allocated.
- Return type:
int
- is_running()[source]¶
Whether there are trials still running.
Generally, if the runner is serial, launching a trial instantly returns its result. On parallel runners, there might be pending configurations to complete.
- Return type:
bool
- iter_results()[source]¶
This method returns any finished configuration, and returns a list with the results of executing the configurations. This class keeps populating results to
self._results_queue
until a call toget_finished
trials is done. In this case, the self._results_queue list is emptied and all trial values produced by running run are returned.- Returns:
A list of TrialInfo/TrialValue tuples, all of which have been finished.
- Return type:
Iterator[tuple[TrialInfo, TrialValue]]
- run(config, instance=None, budget=None, seed=None, **dask_data_to_scatter)[source]¶
Runs the target function with a configuration on a single instance-budget-seed combination (aka trial).
- Parameters:
config (Configuration) – Configuration to be passed to the target function.
instance (str | None, defaults to None) – The Problem instance.
budget (float | None, defaults to None) – A positive, real-valued number representing an arbitrary limit to the target function handled by the target function internally.
seed (int, defaults to None)
- Return type:
tuple
[StatusType
,float
|list
[float
],float
,dict
]- Returns:
status (StatusType) – Status of the trial.
cost (float | list[float]) – Resulting cost(s) of the trial.
runtime (float) – The time the target function took to run.
additional_info (dict) – All further additional trial information.
- submit_trial(trial_info, **dask_data_to_scatter)[source]¶
This function submits a configuration embedded in a
trial_info
object, and uses one of the workers to produce a result locally to each worker.The execution of a configuration follows this procedure:
The SMBO/intensifier generates a TrialInfo.
SMBO calls submit_trial so that a worker launches the trial_info.
submit_trial internally calls
self.run()
. It does so via a call to run_wrapper which contains common code that any run method will otherwise have to implement.
All results will be only available locally to each worker, so the main node needs to collect them.
- Parameters:
trial_info (TrialInfo) – An object containing the configuration launched.
dask_data_to_scatter (dict[str, Any]) – When a user scatters data from their local process to the distributed network, this data is distributed in a round-robin fashion grouping by number of cores. Roughly speaking, we can keep this data in memory and then we do not have to (de-)serialize the data every time we would like to execute a target function with a big dataset. For example, when your target function has a big dataset shared across all the target function, this argument is very useful.
- Return type:
None