Stopping Criteria

In addition to the standard stopping criteria like number of trials or wallclock time, SMAC also provides more advanced criteria.

Termination Cost Threshold

SMAC can stop the optimization process after a user-defined cost was reached. In each iteration, the average cost (using average_cost from the run history) from the incumbent is compared to the termination cost threshold. If one of the objective cost is below its associated termination cost threshold, the optimization process is stopped. Note, since the average_cost method is used, all instance-seed-budget trials of the incumbent are considered so far. In other words, the process can be stopped even if the incumbent has not been evaluated on all instances, on the highest fidelity, or on all seeds.

scenario = Scenario(
    ...
    objectives=["accuracy", "runtime"],
    termination_cost_threshold=[0.1, np.inf]
    ...
)

In the code above, the optimization process is stopped if the average accuracy of the incumbent is below 0.1. The runtime is ignored completely as it is set to infinity. Note here again that SMAC minimizes the objective values.

Automatically Stopping

Coming in the next version of SMAC.